Economics of Peer Review at NSERC
Yesterday, CanadaResearchFunding.org blogged about a paper in a recent issue of Accountability in Research [16(1):13-40] written by Richard Gordon (University of Manitoba) and Bryan Poulin (Lakehead University) which suggests that it would be less costly to fund every qualified NSERC applicant than carry out the peer review process.
The abstract:
“Using Natural Science and Engineering Research Council Canada (NSERC) statistics, we show that the $40,000 (Canadian) cost of preparation for a grant application and rejection by peer review in 2007 exceeded that of giving every qualified investigator a direct baseline discovery grant of $30,000 (average grant). This means the Canadian Federal Government could institute direct grants for 100% of qualified applicants for the same money. We anticipate that the net result would be more and better research since more research would be conducted at the critical idea or discovery stage. Control of quality is assured through university hiring, promotion and tenure proceedings, journal reviews of submitted work, and the patent process, whose collective scrutiny far exceeds that of grant peer review. The greater efficiency in use of grant funds and increased innovation with baseline funding would provide a means of achieving the goals of the recent Canadian Value for Money and Accountability Review. We suggest that developing countries could leapfrog ahead by adopting from the start science grant systems that encourage innovation.”
Some other science bloggers have picked up the story: A Blog Around the Clock, Sandwalk, and Genomicron. I have read the comments on these blogs with interest.
I haven’t been able to get my hands on a copy of the paper yet, so I am not sure if I am ready to provide a fully informed option of their findings. I would really like to know how they calculate the $40,000 “cost of preparation for a grant application and rejection by peer review”.
Basically, I want to know whether that figure is solely the costs once the application is received by NSERC or includes the costs of preparing an average application by the individual applicants.
It is one of those lifecycle questions in economics – where is the starting point?
From a selfish (as a freelance grant writer) perspective, I think that abolishing the grant writing and peer review process would be bad for my business. From a less biased perspective, I think there are other more valid reasons why the peer review process is important. Why? When working as a university grants officer, I saw every NSERC grant application that came out of our institution. Although I’m sure you don’t believe it, I read every single one. I read them for many different reasons – primarily looking for financial, ethical and other concerns from an institutional perspective.
Based on what I saw, I think the inference that “control of quality is assured through university hiring, promotion and tenure proceedings, journal reviews of submitted work, and the patent process” is flawed.
I could tell quite quickly that some applications would never be funded. Do you think that Professor PhD would be very appreciative of being told that they would not be funded by little miss research administrator? I often had long discussions with researchers about their applications and how they could improve them. I hate to generalize, but the egos of applicants were often inversely proportional to their willingness to take feedback. It never shocked me when they were rejected. My anecdotal memories of these instances include researchers at all stages in their careers.
My comments weren’t always tied to technical or organizational aspects of the proposals. My knowledge of auditing research projects made it possible to identify weak or poorly thought out experimental designs and suggest ways to close the gaps, particularly in biological sciences.
The peer review process is especially important for new researchers. Developing a research strategy is a learned skill. Without peer feedback, what will stop these novice researchers from traveling down the garden path? If there isn’t some checkpoint before research begins, how will these individuals obtain objective insight into their research designs?
Peer review is mentoring and networking – even when it is tied to research funding. The challenge becomes – how do we quantify the economic impact of the indirect outcomes of peer review?
Your readers may want to read this themselves. -Dick Gordon, gordonr@cc.umanitoba.ca
[...] writing the post “Economics of peer review at NSERC”, I decided to email the author and direct his attention to the blog discussions on his paper. I [...]
Indeed, the cost of preparing the application includes the estimated time it takes to prepare the application, for both the lead investigator and students/technicians involved.
The paper is certainly worth reading, as it poses an interesting challenge to the current system. Perhaps abolishing the peer-review granting system is a bit extreme, but the fact that it is more expensive to prepare and reject applications than it would be to fund them all gives pause for thought. I agree that you would likely see a spike in applications if investigators knew they had a 100% chance of being funded, but perhaps a better system is out there. The suggestion of providing a 30K baseline of funding for new researchers is certainly an interesting idea. If it is cheaper to do so, as the authors suggest, it is certainly an innovative idea to provide all new researchers with a baseline of funding for their first five years, thereby providing a modicum of stability during those first years when they are innovative, but inexperienced and possibly overwhelmed with the practicalities of being a junior faculty member.