I have had a bit of difficulty getting clarifications on my questions about the binning process and in fact was not able to vet my interpretation through NSERC as I had hoped. But rather than wait for official clarification, I would like to post my opinion and interpretation of official documents already available on the web. My resources for this activity included:
My initial challenge was understanding how the NSERC Evaluation Groups (EGs) will bin the proposals and assign an evaluation indicator of ‘high’, ‘normal’ or ‘low’ to the relative cost of research. The definitions of these terms are:
- High: Majority of justified expenses represent costs higher than the norm for the research area
- Normal: Majority of justified expenses are within the norm for the research area
- Low: Majority of justified expenses are lower than the norm for the research area
Initially, I took this to mean that if the budget was higher than the norm for the area of research, then it would be scored as high and that would have a detrimental impact on the overall scoring of the proposal. I also inferred that there would be a similar effect for a low scoring. From various unofficial conversations and feedback from my colleagues, my understanding now is that this step will rate the application based on the relative cost of research and this separate qualification will not impact the overall ranking of the application.
The take home here is: make sure to ask for what you need in your budget and be clear in your justification why and how these expenses are required in the context of the normal costs of research in your field. (For example, if no one in your field asks for a technician and you are – then be sure to justify why this higher cost activity is necessary.)
Once the application is ranked based on ‘Excellence of the Researcher’, ‘Merit of the Proposal’ and ‘Training of HQP’, the relative cost of research rank (or bin) will qualify the proposal for an increment adjustment to the normal cost of research for the relative discipline. To clarify this further, if the normal cost of research in a given field is $30,000 and a proposal is qualified with a high cost of research, it would qualify for an increment adjustment of $7,000, netting that particular applicant an award of $37,000 per year. (These numbers are generated by me and do not reflect any practical examples.) This step alone is probably responsible for the DG2009 researchers that found themselves adjusted sharply downwards. I am aware of at least one researcher whose grant amount was decreased to 51% of the previous award, in a field that typically has a much lower average cost of research. From what I understand so far, I suspect this system does not have a mechanism to take into account those individuals whose cost of research is significantly higher than the norm or even the adjusted high cost of research amount (e.g. CRCs or other high profile researchers). Somehow this just seems counterintuitive to the S&T strategy to fund everyone in a particular field at the same level – academic research isn’t preschool where everyone needs to be treated equally.
The ranking of proposals step is effectively the same as it was before, except now EG members are being given this matrix to help them with their analysis. The addition of the relative cost of research qualifier to the decision making process is the major change. It should also be noted that the step of ranking the proposal and qualifying the cost of research is completed by the specific EGs, but the step of actually assigning a funding level will be undertaken by the Executive Committee of the EGs (consisting of the Section Chairs and Group Chairs).
So then the question becomes, how does NSERC set the normal cost of research for individual disciplines? From the FAQ:
NSERC already uses studies of research costs to earmark funding for First-Time Applicants in each GSC. The intent is to broaden the approach to set the budgets for GSCs in the future. Currently, NSERC is using information from various sources including Statistics Canada and grant recipient expenditure data, and is comparing this with data from the United States. The new methodology, when ready, will be validated by the Committee on Grants and Scholarships (COGS), the standing committee of NSERC that oversees the Discovery Grants program.
The rationale for taking the applicant’s previous grant award level out of the funding picture, from the FAQ:
An applicant’s funding history plays too much of a role in the awarding of a new grant. Funding is often incremental and new grants within a few thousand dollars of the previous grant. This has resulted in a level of inertia that is too high in the program overall. The new model preserves stability and continuity of funding for researchers who maintain a strong record of contributions to research and training, but it also permits a rapid ramp up of funding for applicants with exceptional contributions, no matter their history in the program.
I can’t help but think that this will just result in a future level of inertia if NSERC is constantly setting the bar for the “normal cost of research” funding amount and the increments for adding to “high proposals” or subtracting from “low proposals”. This is just an observation since I don’t really have a solution to the inertia challenge either. I know that this will help NSERC stabilize annual budget requests, but I don’t really see how it can respond dynamically to the needs of our innovation economy.
Ruminations aside, from the peer review manual, for your reference:
6.8.4. Relative Cost of Research
In addition to the selection criteria discussed previously, applications may be assessed with regard to the relative cost of research. The “Cost of Research” relates to individual circumstances, but in the context of an area of research. Evaluation Groups will collectively determine the parameters for considering the cost of research. Members will be asked to first gauge the budget in terms of justification, and then to rate the relative cost of the proposed research program (High, Normal, Low) as compared to the norm for the research areas represented in the applications considered by the Evaluation Group(s).
[snip]
Using the indicators: The evaluation indicators suggest statements for a High, Normal and Low cost of research factor. The relevance of these will vary by field. Members’ understanding of the norms for the research areas represented in the applications considered by the Evaluation Group is an important complement to the indicators. A well-justified budget that outlines the need to do field work in the Arctic may be rated as having a cost of research higher than the norm.
6.8.4.1.1. Points of Reflection
Appropriateness of, and justification for, the budget – The justification for the funding requested is considered within the Cost of Research. In the instructions, applicants for a Discovery Grant are asked to prepare a realistic budget. An Evaluation Group member may determine that the amount requested is higher than what is justified by the research proposal. If a budget is deemed to be inflated, members are asked to assess the relative cost of research in consideration of what is felt to be a more realistic budget. Applicants are not to be penalized for providing what is perceived as an inflated budget (i.e., they should not automatically receive an assessment of the relative cost as Low). However, applicants should consider that presenting an inflated budget makes it difficult for members to respect any claims for particularly high costs associated with the proposed research.
Relationship to Other Sources of Funding – A Discovery Grant may not represent a researcher’s only, or even major, source of support. The availability of other sources of support should not systematically result in an assessment of the relative cost as Low. As long as the contributions to the NSE field described in the research proposal are commensurate with the funds requested, the availability of other sources of funding should be viewed positively. Evaluation Groups may want to consider the research plan and budget justification relative to the applicant’s capacity to undertake the planned program given other commitments (including research funded through other sources, or in the context of a large research operation), and this can be reflected in the cost of research rating, where appropriate. The applicant should demonstrate sufficient time and resources to commit to the proposed research program.
