You may not be aware of it, but NSERC is currently in the middle of an important exercise which may ultimately lead to a redistribution of their grant funds among disciplines. A group called the Council of Canadian Academies is conducting an assessment of “performance indicators for basic research.” While the wording on the NSERC web page that explains this assessment is not exactly easy to parse, my interpretation of it is that NSERC wants to know what tools they can use to quantitatively assess the bang they are getting for their buck in the different fields that they fund. The implication is that they will use whatever tools they come up as a result of this assessment to carry out a reallocation of funds among the disciplines.

Now, we are all scared of changes at NSERC, but this is not necessarily a bad thing. For one thing, I was intimately involved in NSERC’s last reallocation process, and even though condensed matter physics came out very much ahead of the curve in that exercise, I can assure you that the process was severely flawed. And there is nothing wrong with a government wanting to spend its money in the most cost-effective way. Besides, we all know that our work is so much better and more important than everyone else’s, so it is self-evident that any measure of the value of our work relative to anyone else’s will have to work in our favor, right? Sorry, but “self-evident” won’t cut it. NSERC is looking for “scientific” ways of measuring the relative value of, say, Astronomy vs. Chemical Engineering.

How might this work? First, how do we gauge the impact of one individual’s research? We all know that we shouldn’t just count papers when we looks at someone’s CV, but we all do it anyway. However it is easy for someone to publish a lot of mediocre papers, or a lot of papers on a subject that no-one else is interested in, so while the number of papers alone might be taken as some sort of measure of an individual’s productivity, it is not a useful measure of the quality or the impact of the product. 

The h-index has become a popular measure of an individual’s impact. In case you don’t know, your h-index is X if you have X papers that have been cited at least X times (Hirsch, PNAS 102, 16569 (2005)). For example, my h-index is 21, meaning that 21 of my publications have been cited 21 or more times. My other 60-odd papers have been cited fewer than 21 times, including a disturbing number – some of which I think are actually quite good – that have never been cited at all.  In my field, an h-index of 20-ish is reasonable at my career stage – probably above average, but certainly not at the top of the heap. And, yeah, that sounds about right to me as a measure of my impact in the field. One of the problems with the h-index, though, is that it is very discipline-dependent. In Mathematics, for example, a typical paper might have a bibliography containing four references, whereas mine usually have forty! I found some data in a preprint posted on arXiv (Iglesias and Pecharroman, arxiv.org/ftp/physics/papers/0607/0607224.pdf) indicating that on average, a paper is cited about 7.2 times per year in physics, compared to 4.3 times in materials science, 2.6 times in mathematics, and 24.6 times in molecular biology (sorry, no data for astronomy). One can infer from this that an h-index of 21 would be world-class in math, and laughable in genetics. In fact, the paper by Iglesias and Pecharroman is about a normalized index which takes these disciplinary variations into account.

 On a grander scale, how do we gauge the impact of an entire field of research within a given country? This is really what NSERC wants to know. There are people who spend a lot of time thinking about this, and a number of quantitative indices have been developed. For example, one can take the citations received per year by all the papers published by Canadian physicists, and divide it by the world average citations per year in the same field. If the resulting number is bigger than 1, then Canadian papers are cited more frequently than the world average in this field. There are also measures based on the impact factor (number of citations/number of papers published) of the journals in which the papers are published – so if everyone in a certain country publishes in rinky-dink journals, that country gets a low score, while if everyone publishes in Nature, they get a high score.  You can get some information about this field (called bibliometrics) at www.science-metrix.com, which is the web site of a company that does this stuff in a very serious way.

I understand NSERC’s desire to get some intelligent guidance as to where they should put their money, and I am sure that NSERC plans to rely on more than just a set of numbers to inform any decisions they make about reallocation. I am also all for an informed, scientific approach to just about anything. I confess, however, that I am a bit leery of such a quantification of the research enterprise.  The fact is that numbers tend to dominate the way we think about such things, and any other available information becomes secondary. It is much easier to make a decision based on whether a particular number is above or below some criterion, than it is to carefully weigh a large amount of qualitative and disparate information.  Besides, even if these bibliometric indicators are robust and reliable (which they probably are, since they involve a lot of averaging), reducing the combined efforts of hundreds of smart, creative, hardworking people down to a single number seems to me somewhat insulting.