In 2010, NSERC asked the Council of Canadian Academies to look into ways to assess research performance in the various NSERC disciplines. The resulting CCA report has now been released.

Here is a link:

http://www.scienceadvice.ca/en/assessments/completed/science-performance.aspx

This web page gives a brief summary of what’s in the report. You can also download the full report there if you want to. The full report is 162 pages long, and no, I haven’t read it yet.

At first glance, thoiugh, it appears to me that the conclusions of the report are reasonable. Here is a summary, taken from the above web page:

“After examining the available evidence, the Council’s Expert Panel came to a small number of key findings, listed below, and concluded overall that quantitative indicators should inform, rather than replace, expert judgment in science assessment for research funding allocation. The Expert Panel also developed four guiding principles to support research funding agencies undertaking science assessments in support of budget allocation, they are: context matters; do no harm; transparency is critical; and expert judgment remains invaluable. The principles are expanded upon with the Panel’s report.”

The “quantitative indicators” they are referring to are things like bibliometric measures, as discussed in one of my earlier posts.

Here are their “key findings:”

• Many science indicators and assessment approaches are sufficiently robust to be used to assess science performance at the field level in natural sciences and engineering (NSE).
• Quantitative indicators should inform, rather than replace, expert judgment in science assessment for research funding allocation.
• International “best practices” offer limited insight into science indicator use and assessment strategies.
• Mapping research funding allocation directly to quantitative indicators is far too simplistic, and is not a realistic strategy.
• There is no compelling reason for certainty that past successes will lead to future successes or past failures to future failures. As a result, science indicators may not always provide a reliable guide to future prospects.

My understanding is that NSERC will give the research community an opportunity to respond to this report, although as yet we don’t know any details. As a member of CAP’s NSERC Liaison Committee, I will be involved in the physics part of this response, so feel free to pass your comments on to me. I also understand that NSERC will then take this report into consideration in deciding on a new Discovery Grant “reallocation process” to move money around among the disciplines. This means that it’s potentially quite important.