Consultations

CCA Report

Assessment by Council of Canadian Academies on Science Performance and Research Funding

On behalf of NSERC, the Minister of Industry has asked the  Council of Canadian Academies (CCA) to conduct an assessment on performance indicators for basic research. The CCA has published the expert panel’s report,  Informing Research Choices: Indicators and Judgment. The assessment examines approaches used to evaluate research performance and indicators that enable comparisons across areas of research in the natural sciences and engineering.

This is part of NSERC’s ongoing commitment to evaluate and improve its processes. The information may help NSERC find ways to compare overall levels of excellence across disciplines, according to international best practices. These and other potential considerations, such as indicators of research trends and research capacity across disciplines, may help determine appropriate budget allocations between the various Discovery Grants Evaluation Groups in the future.

FAQs

What are NSERC’s plans now that the CCA has published the expert panel`s report?

NSERC will invite feedback from the research community. Consultation details will be made available on NSERC’s Web site. The information contained in the expert panel’s report as well as feedback from the research community will contribute to the goal of establishing a reliable methodology to allocate budgets between Evaluation Groups under the Discovery Grants Program. These plans are not expected to affect the next Discovery Grants competition to be held in 2013. NSERC aims to implement any changes to the budget allocation methodology in time for the 2014 Discovery Grants competition.

What specific question does this assessment address?

The assessment addresses one overarching question and three sub-questions:
What do the scientific evidence and the approaches used by other funding agencies globally have to offer, in terms of performance indicators and related best practices in the context of research in the natural sciences and engineering, carried out at universities, colleges and polytechnics?

1. What existing qualitative and quantitative indicators and metrics are relevant to budget allocation in the context of support for research in the natural sciences and engineering, and how can they be categorized (e.g., “shelf life”; cross-disciplinary and international comparability; relevance to interdisciplinary vs. focused disciplinary areas; and applicability to emerging vs. established research areas)?
   
   In this context, quantitative indicators include researcher population dynamics and cost-based indices, while qualitative indicators include collective quality/excellence of research and areas of strength indices.
   
   
2. What are international best practices in the construction, methodological review, and use of quantitative and qualitative indicators for research evaluation and budget allocation in support of basic research in the natural sciences and engineering?
   
   
3. Considering the foregoing, and in light of the Government of Canada’s Science and Technology Strategy and NSERC’s objectives for the support of research, what key considerations (e.g., risks, advantages/disadvantages, behavioural and institutional consequences) and principles emerge in determining defensible use and balance/weighting of performance indicators/metrics for budget allocation?

What is the rationale for this assessment?

• For the Discovery Grants (DG) program, the budget allocation between disciplines is weighted toward demand-driven historical precedents; it should better account for other factors in the evolution of the research environment, such as the changing demographics of the disciplines, differences in cost structure across disciplines and areas of strength and opportunity.
• The challenge for NSERC, and other research funding agencies, lies in identifying appropriate and robust indicators, as well as available metrics, to measure and compare research quality, research trends and research capacity across disciplines.
• There is a knowledge gap—the scholarly literature and practices elsewhere offer no single methodology applicable to the Canadian context and to the DG program.

How might this assessment affect the amounts of individual or team grants?

• The process for peer review of individual or team grant applications is outside the scope of this assessment; however, the overall budget available for a given discipline will have an impact on average grant sizes and success rates.

Why was the assessment needed now?

• The timing for this assessment follows two reviews of the DG program and the termination of the Reallocation Exercise used from 1994 to 2007.
• This assessment addresses a concrete and timely need. NSERC requires a new mechanism to systematically review and update the allocation of funds between  discipline clusters.

How will this assessment help fix the problem?

• In terms of impact, a CCA assessment of performance indicators for basic research discusses potential methodologies and indicators and assesses their relevance to enable comparisons across disciplines and areas of research.
• These potential methodologies and indicators, once validated, will improve NSERC's allocation of $350 million per year in research support, and make the program more responsive to an evolving environment (a federal S&T Strategy commitment).
• NSERC will use the assessment to inform decisions on the methodology to allocate funds across areas of research in support of science and engineering.
• The assessment is intended to enable policy development, rather than prescribe policy. In other words, the expert panel provides lists of possible indicators, methodological guidelines and general principles, not a prescribed methodology for their application to budget allocation.

Why was the CCA asked to conduct this assessment?

• A Canadian assessment is required to inform Canadian policy.
• NSERC could not conduct this analysis for reasons of capacity.
• A CCA assessment was sought because it is critical that NSERC’s methodology is informed through a transparent process (e.g., publicly available assessment report) and based on principles that have been validated through scientific input.
• Confidence that indicators are valid and robust is required in the light of the risks of creating unintended distortions in the allocation process.  
• A CCA assessment will enable NSERC to build on available scientific evidence.

How is this assessment science-based?

• It is informed by the scientific literature; in particular, scholarly literature on the use of science indicators and assessment methodologies for informing research funding allocation in the natural sciences and engineering.
• It addresses current knowledge gaps in relation to:
  • applying the existing knowledge to the Canadian context;
  • the validity and appropriate uses of the large number of existing S&T metrics.

What was the assessment expected to deliver?

• A catalogue/library of indicators to measure and compare needs/performance across areas of research and an assessment of their relevance, applicability and “shelf-life” for use in budget allocation;
• A contextual piece on best practices in research evaluation and budget allocation with overall guiding principles and criteria for the selection of indicators, as well as an evaluation of a set of practices and indicators;
• In light of current knowledge and experience and of NSERC's objectives for support for basic research, a discussion of the key considerations (risks, advantages/disadvantages, behavioural and institutional consequences) and principles that emerge in determining defensible use and balance/weighting of performance indicators/metrics in the context of a budget allocation process.

Why does the assessment question refer to colleges and polytechnics when the Discovery Grants Program is primarily aimed at university researchers?

• Posing a broader question draws from a potentially wider set of approaches and allows for the possibility of applying the findings to other NSERC programs, including funding that involves colleges and polytechnics.