I have had a bit of difficulty getting clarifications on my questions about the binning process and in fact was not able to vet my interpretation through NSERC as I had hoped. But rather than wait for official clarification, I would like to post my opinion and interpretation of official documents already available on the web. My resources for this activity included:

• The Peer Review Manual, updated on September 28, 2009, specifically Chapter 6;

• The GSC Structure Review FAQ;
• as well as various informal conversations.

My initial challenge was understanding how the NSERC Evaluation Groups (EGs) will bin the proposals and assign an evaluation indicator of ‘high’, ‘normal’ or ‘low’ to the relative cost of research. The definitions of these terms are:

- High: Majority of justified expenses represent costs higher than the norm for the research area
- Normal: Majority of justified expenses are within the norm for the research area
- Low: Majority of justified expenses are lower than the norm for the research area

Initially, I took this to mean that if the budget was higher than the norm for the area of research, then it would be scored as high and that would have a detrimental impact on the overall scoring of the proposal. I also inferred that there would be a similar effect for a low scoring. From various unofficial conversations and feedback from my colleagues, my understanding now is that this step will rate the application based on the relative cost of research and this separate qualification will not impact the overall ranking of the application.

The take home here is: make sure to ask for what you need in your budget and be clear in your justification why and how these expenses are required in the context of the normal costs of research in your field. (For example, if no one in your field asks for a technician and you are – then be sure to justify why this higher cost activity is necessary.)

Once the application is ranked based on ‘Excellence of the Researcher’, ‘Merit of the Proposal’ and ‘Training of HQP’, the relative cost of research rank (or bin) will qualify the proposal for an increment adjustment to the normal cost of research for the relative discipline. To clarify this further, if the normal cost of research in a given field is $30,000 and a proposal is qualified with a high cost of research, it would qualify for an increment adjustment of $7,000, netting that particular applicant an award of $37,000 per year. (These numbers are generated by me and do not reflect any practical examples.) This step alone is probably responsible for the DG2009 researchers that found themselves adjusted sharply downwards. I am aware of at least one researcher whose grant amount was decreased to 51% of the previous award, in a field that typically has a much lower average cost of research. From what I understand so far, I suspect this system does not have a mechanism to take into account those individuals whose cost of research is significantly higher than the norm or even the adjusted high cost of research amount (e.g. CRCs or other high profile researchers). Somehow this just seems counterintuitive to the S&T strategy to fund everyone in a particular field at the same level – academic research isn’t preschool where everyone needs to be treated equally.

The ranking of proposals step is effectively the same as it was before, except now EG members are being given this matrix to help them with their analysis. The addition of the relative cost of research qualifier to the decision making process is the major change. It should also be noted that the step of ranking the proposal and qualifying the cost of research is completed by the specific EGs, but the step of actually assigning a funding level will be undertaken by the Executive Committee of the EGs (consisting of the Section Chairs and Group Chairs).

So then the question becomes, how does NSERC set the normal cost of research for individual disciplines? From the FAQ:

NSERC already uses studies of research costs to earmark funding for First-Time Applicants in each GSC. The intent is to broaden the approach to set the budgets for GSCs in the future. Currently, NSERC is using information from various sources including Statistics Canada and grant recipient expenditure data, and is comparing this with data from the United States. The new methodology, when ready, will be validated by the Committee on Grants and Scholarships (COGS), the standing committee of NSERC that oversees the Discovery Grants program.

The rationale for taking the applicant’s previous grant award level out of the funding picture, from the FAQ:

An applicant’s funding history plays too much of a role in the awarding of a new grant. Funding is often incremental and new grants within a few thousand dollars of the previous grant. This has resulted in a level of inertia that is too high in the program overall. The new model preserves stability and continuity of funding for researchers who maintain a strong record of contributions to research and training, but it also permits a rapid ramp up of funding for applicants with exceptional contributions, no matter their history in the program.

I can’t help but think that this will just result in a future level of inertia if NSERC is constantly setting the bar for the “normal cost of research” funding amount and the increments for adding to “high proposals” or subtracting from “low proposals”. This is just an observation since I don’t really have a solution to the inertia challenge either. I know that this will help NSERC stabilize annual budget requests, but I don’t really see how it can respond dynamically to the needs of our innovation economy.

Ruminations aside, from the peer review manual, for your reference:

6.8.4. Relative Cost of Research
In addition to the selection criteria discussed previously, applications may be assessed with regard to the relative cost of research. The “Cost of Research” relates to individual circumstances, but in the context of an area of research. Evaluation Groups will collectively determine the parameters for considering the cost of research. Members will be asked to first gauge the budget in terms of justification, and then to rate the relative cost of the proposed research program (High, Normal, Low) as compared to the norm for the research areas represented in the applications considered by the Evaluation Group(s).

[snip]

Using the indicators: The evaluation indicators suggest statements for a High, Normal and Low cost of research factor. The relevance of these will vary by field. Members’ understanding of the norms for the research areas represented in the applications considered by the Evaluation Group is an important complement to the indicators. A well-justified budget that outlines the need to do field work in the Arctic may be rated as having a cost of research higher than the norm.

6.8.4.1.1. Points of Reflection

Appropriateness of, and justification for, the budget – The justification for the funding requested is considered within the Cost of Research. In the instructions, applicants for a Discovery Grant are asked to prepare a realistic budget. An Evaluation Group member may determine that the amount requested is higher than what is justified by the research proposal. If a budget is deemed to be inflated, members are asked to assess the relative cost of research in consideration of what is felt to be a more realistic budget. Applicants are not to be penalized for providing what is perceived as an inflated budget (i.e., they should not automatically receive an assessment of the relative cost as Low). However, applicants should consider that presenting an inflated budget makes it difficult for members to respect any claims for particularly high costs associated with the proposed research.

Relationship to Other Sources of Funding – A Discovery Grant may not represent a researcher’s only, or even major, source of support. The availability of other sources of support should not systematically result in an assessment of the relative cost as Low. As long as the contributions to the NSE field described in the research proposal are commensurate with the funds requested, the availability of other sources of funding should be viewed positively. Evaluation Groups may want to consider the research plan and budget justification relative to the applicant’s capacity to undertake the planned program given other commitments (including research funded through other sources, or in the context of a large research operation), and this can be reflected in the cost of research rating, where appropriate. The applicant should demonstrate sufficient time and resources to commit to the proposed research program.

The following is a non-exhaustive list of rules of thumb for budgets (hopefully I don’t get skewered for putting this in print):

Equipment

• a computer purchase can be estimated at ~$2.5k

• Don’t forget to include the cost of federal and provincial taxes in your purchase – don’t let yourself be blindsided by these – they are real and justifiable parts of your cost (most academic institutions only pay a proportion of the GST, therefore use the unrebated portion of the tax in your calculations)
• If you have a large piece of equipment in your lab, you can ask for peripherals to expand/extend use (i.e. lenses for a microscope, etc.)

Materials & Supplies

• Life science disciplines, multiply the number of people supported by the grant by $10k (i.e. 2 grad students and a half time technician = $25k request) adjust downward if you don’t have enzyme or animal costs

• For grants involving cellular, molecular, developmental, biomedical, etc. research, assume an average real cost of ~ $1.5k per month ($18k per year) in supplies
• If you plan to hire undergraduate researchers (summer or otherwise) plan for additional materials and supplies expenses (for wastage)

Travel

• $4-5k for the PI, $1-2k per each supervised student/post-doc

• Aim for about 5% of total budget request

Dissemination

• Estimate ~$1k per paper, unless you use open source publications, then estimate ~$2.5k per paper

Other

• $200-1k for the cost of miscellaneous items such as photocopying, faxing, printing, toner cartridges, etc.

Do you have other rules of thumb to share? Please feel free to add by commenting on this post or sending me an email. I’ll post a more comprehensive list as material becomes available.

(Note: This post was originally posted in October 2008, but I’m not above recycling good information when I’m short on time. It is slightly updated. Hope you find it useful!)

Last year, Izabella Laba described how an average Discovery grant in mathematics is spent in her blog. This discussion is very interesting as it gives some perspective on how challenging it can be to fund research in mathematics.

I posted this last year, but I still think that it provides some relevant insight into some of the challenges that math and stats researchers face. If you have the time, Dr. Laba also discusses other aspects of her experience securing enough funding to run a research program – it’s worth a read.

If you don’t ask, you can’t get.

While the pressure to stay within the “norm” is increasing in this competition, it is important for applicants to remember that it is just as dangerous to lowball research expenses as it is to ask for too much. The key element to remember is that the budget and the research plan need to be linked. If the proposal includes projects that require 5 pairs of (graduate student) hands to meet the workload demands and the budget only asks for funds for one student (with no discussion of how the other four bodies will enter the project) – the applicant has effectively demonstrated that they do not know how to plan or manage research.

Many applicants devote all of their energy to writing the research plan without any thought towards the budget until the last few days before the deadline, then they ask their colleagues what the norm is for their field and plug that data into their budget forms. If you plan for a Porche and budget for a Smart Car, you are either going to be driving a Smart Car or not driving at all. The new scorecard system for binning applicants based on their budget request is going to make planning an appropriate budget an increasingly important element of Discovery Grant applications.

So how do you find out the norm? Ask colleagues, look at the statistical analysis of previous Discovery Grant competitions (links to: 2009 data, 2008 data), search through the NSERC award database to find out what competitive labs are receiving and then make a budget that is closely linked to the research plan.

While the GSGs are no more, looking up the min/average/max award amounts for the most relevant GSGs will help provide a ballpark of the normal funding levels for a particular field of study. This information is available from either the statistics links above or the NSERC Awards Search Engine. Some departments and/or research offices also keep records on the norm and might provide an opportunity to find out the budget breakdown in addition to the budget totals. It can’t hurt to ask – you never know what data might be shared.

While knowing the total award doesn’t tell the entire story and shouldn’t dictate the budget to apply for, it does give a guideline when writing a research plan to ensure that the plan (and linked budget) fall within a normal funding range for a particular field. This information is most useful for new applicants who are uncertain how to set their first budget, but is a good touchstone for returning applicants as well.

From the NSERC DG2010 Information Session slides & the DG2009 statistics:

To give a bit more context to the previous post.