rss

Economics of peer review at NSERC – a pernicious discussion

After writing the post “Economics of peer review at NSERC”, I decided to email the author and direct his attention to the blog discussions on his paper. I was thrilled by Gordon’s rapid response and provision of the paper in question:

Gordon, R., & Poulin, B. (2009). Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant Accountability in Research, 16 (1), 13-40 DOI: 10.1080/08989620802689821

Gordon has also become active in the blog discussions of the paper in: A blog around the clock, Genomicron, Pyrenaemta, and Sandwalk. There is also more discussion on Bora Zivkovic’s (A blog around the clock) newest post on the topic and friend feeds, here and here, although Gordon hasn’t been present yet. One of the most interesting things about this discussion (to date) is the observation that most, if not all, of the posters and commentators didn’t read the paper prior to publishing their opinions. (I acknowledged in my post that I was commenting solely on the abstract, while many others did not.) I am not really planning to talk about the web 2.0 and social media implications of this further, but I think that it is a very interesting point to note.

One of the commentators on Bora Zivkovic’s original blog on the topic, his brother Marko in fact, wrote:

“ . . . I want to address something I have seen in the comments here (as well as at Lawrence Moran’s Sandwalk blog) that I think is highly pernicious. I find it really sad that scientists themselves would interiorize the highly insulting view that were we not forced to write grants we would just slip into crappy, useless research. This is precisely the attitude implied by the granting agencies: if we don’t slap ridiculous amount of self-justification and scrutiny onto you, you will take the money and run to the Bahamas. I think it is highly damaging to science to consider its practitioners as extrinsically motivated (some sort of Skinner’s pigeons who perform only when given reinforcement).

I would say that science types are by and large a self-selected bunch who obviously are not motivated by mere monetary rewards (if they were they would have turned their supposedly powerful brains to more lucrative endeavors such as business, law, etc.). Second, they pass through a long and arduous apprenticeship that culminates in a PhD. By then they should have soaked up the scientific ethos thoroughly. In addition, throughout our career trajectories we are scrutinized most carefully — when we get the job, when we are promoted, when we publish, etc. The article points out, and I think quite rightly, that scientists are judged more rigorously at all these points and that the one area where they should be trusted is precisely when they are embarking on new research.

. . . There will be plenty of “quality control” later on in the process (publishing, etc.). We as scholars/researchers/scientists are intrinsically motivated to search for truth, or the best approximation to it, with all we have and in the best way we could conceive of. We are the types who would do it even if we had just a pen and a piece of paper. We are like Picasso who supposedly said that if he were put in a solitary cell with nothing to draw with he would paint by spitting on the walls.”

I suppose that I am guilty of propagating this viewpoint, however, I think that it is dangerous to generalize about any population. Yes, the vast majority of science types (or academic types, for that matter) are highly motivated and curiosity-driven - I have met and worked with many shining examples of these individuals. Unfortunately, I have also crossed paths with some less savory types and had to clean up their messes. While they are the minority, the damage they wreak is not inconsequential and it has the unfortunate side effect of catching and staying in the public psyche – à la Scott S. Reuben (for example).

The reality is that incidents of research and budgetary fraud have created an environment where administrative controls are more prevalent. The explosion of administrivia is a direct consequence of additional reporting requirements and strict enforcement rules required by funding agencies. I don’t believe that there is a precedent, but many institutions live in fear of losing all agency money because of the whims of one shyster.

Attend a CAURA (Canadian Association of University Research Administrators) meeting and you’ll hear some shocking stories of incidents stretching the gamut of research fraud, grad student mistreatment, creative budgeting, ethics, and safety issues. The controls won’t stop the malicious fraudsters, but the burdens they place on the academic community as a whole are there for a reason. Sorry, trust isn’t always possible.

How does this relate back to the paper? Gordon and Poulin suggest that NSERC set aside a proportion of their funding portfolio (i.e. the Discovery grant budget) to provide baseline grants to all researchers. They do not advocate abolishing peer review altogether, but indicate that it would be more cost effective and stimulate more innovative research if each qualified applicant was given a seed grant to spend at their discretion (~$30,000).

There are many aspects of the paper that I agree with and some that I disagree with – it is definitely a thought provoking read. While I can see the benefit of such a plan, I am having a tough time thinking around the complexities of implementation.

What complexities? Well, obviously if you wiped out a significant proportion of the peer review process at NSERC, there would be cost efficiencies (i.e. smaller travel budgets for reviewers) and you certainly wouldn’t need as many program officers (i.e. a leaner NSERC); then those monies could be redirected to fund more research. The challenge comes in at the institutional level – how to administer the funds. Most institutions use grant applications to determine the level of risk that the research plan entails and therefore ensure that appropriate safety checks are put in place. Canadian institutions are more relaxed about budgetary adherence (especially for NSERC grants) versus their US counterparts. I suppose that a one or two page research and budget plan would address this adequately. I suspect that there would be a trickle down of administrative controls and an offloading of that cost line from the federal budget (of course, this might be offset by an increase in indirect costs funding).

Also, if everyone who asked for money got money, how do you handle stagnant researchers? Who decides to take the money away? Researchers have to voluntarily decide that they no longer want (!) to do research and forgo baseline funding? I think that this would create a huge swell of research emeriti and thereby increase the teaching burdens of new faculty by reducing the number of tenure-track teaching positions available. I think that this alone would negate the “savings” in research efficiency by avoiding the grant writing step for this funding.

It also strikes me that this would be a circular model: competitive hiring of a tenure-track faculty member, receipt of a baseline grant, publish, teach, service, promotion, baseline continues . . . et al. etc. . . . tenure achieved, baseline continues. Does baseline ever change? What happens if promotion stalls – does the baseline stop? What happens if this model is applied and found ineffective – how would you ever repeal funding for everyone – I can already feel the groundswell of backlash.

I’m not saying it couldn’t work, but it would require a big paradigm shift not only for researchers, but also for the upper echelon and worker bee levels of administrators to make it happen.

The Best Job in the World

I have become increasingly fascinated with Tourism Queensland’s competition for “The Best Job in the World”. The successful candidate will live on Hamilton Island and will spend their time exploring the Islands of the Great Barrier Reef and reporting on their experiences through video diaries and blog posts. The call for applicants occurred on January 11th and the top 16 candidates were announced on April 3rd.

No, I am not interested in the job itself, I am interested in the concept as a case study in the use of web 2.0, social media, and viral marketing. According to media releases, the competition received over 34,000 video applications. Anthony Hayes, CEO of Tourism Queensland, said,

“In many ways [the competition] has taken on a life of its own spawning special discussion groups, bulletin boards, blogs and websites with applicants critiquing the competition, having detailed discussions and swapping ideas and tips. …. During the first weekend of the campaign . . . more than 200,000 people from around the world logged onto the website, including 25,000 in one hour alone.”

Wow. That is an amazing amount of traffic.

From a March 25th press release: “[t]he worldwide response to Tourism Queensland’s Best Job in the World campaign has been nothing short of phenomenal and already we have achieved around $100 million dollars worth of global publicity for Queensland.”

The top 50 applicants were tasked with campaigning for votes for a wild card spot on the top 10 interview list. Tourism Queensland ended up inviting 16 people, including the wild card, for interviews in early May. The wild card candidate, Clare Wang, received over 150,000 votes.

But get this, the wild card voting process tallied over 470,000 votes.

What fascinates me the most is the fact that Tourism Queensland spawned a global virtual army of social and conventional media marketers through the process of selecting the interview candidates. Yes, that’s right, they haven’t even given anyone a job yet! In the April 3rd press release announcing the top 16, Tourism Minister Peter Lawlor, stated,

“Tourism Queensland has spent the past few weeks watching in amazement at the lengths these candidates have gone to in their quest for The Best Job in the World. From organising stunts like scuba-diving in a tank in Amsterdam, dressing up as a mermaid in downtown Singapore and spruiking at a London pub to conducting hundreds of media interviews, the Top 50 have shown an incredible amount of ingenuity and passion. Within a week of being short-listed, Brisbane applicant Hailey Turner had even organised a 12 day round-the-world trip promoting herself and the Islands of the Great Barrier Reef.”

The process isn’t complete yet as it still remains to be seen whether all this awareness will translate into addition tourism, but as an example of how viral and social media marketing can impact web traffic, I think that “The Best Job in the World” is an unqualified success.

I’m convinced that Tourism Queensland’s experiment would make a fabulous marketing case study. Analyzing what they did right, what they did wrong, how they managed and measured the process, and how you could repeat their results would be a terribly interesting read. Maybe someone with one of those new SSHRC business-focused graduate scholarships could make this their thesis topic.

Now that would be my dream job . . . being paid to do my own curiosity driven research.

Fave me!

The editor in me shudders at the title, but I’m not above blatantly asking for approval. It almost feels like one of those awkward high school moments. . .

Yes, I have joined Technorati.     Add to Technorati Favorites

I recently discovered that some of my posts have been picked up by Technorati so I figured that I better “ride the wave”. Please make me feel popular and fave me. All the cool kids are doing it. Really.

I even faved myself, but Jake Seliger did it first, so I thought it would be okay. Oh no - I just went and gave you more authority Jake!

Taking the time to get the take home

In February, I attended a seminar called “Blogs, podcasts and online syndication” delivered by Dean Owen of BRASS Media through the Alberta Business Link. The Business Link is a not-for-profit business support service provided by the government. They have lots of resources available to entrepreneurs and have a fabulous service called the “Guest Advisor Program”.

I quite like the seminars the Business Link broadcasts through their videoconferencing system. I have attended quite a few of these sessions and find that most of the material is pretty basic, but can be a nice introduction to a wide variety of topics. In fact, their portfolio of offerings has gotten better over the past year. An added incentive is that most of the seminars are free – and you just can’t beat that value!

Sometimes it is difficult to take the time to learn new things when there are a million other things clamouring for your attention. Of course, it is hard to justify going to a course when you aren’t sure if you will actually learn anything – shaking off cynicism can be tough. Keeping in mind Parkinson’s Law, (I’ll paraphrase: ‘work will expand to fill all the time available’) I am trying hard to schedule learning, marketing and exercise commitments into my schedule to ensure that I make time for these important things in my life. A few years ago I decided that I would apply a rule of thumb: if I learnt one new thing per hour from a course or training session then it wasn’t a waste of my time. If I learned two new things in total, the course was a great investment of my time.

But, back to the blogging seminar. . . I learned much more than two things! So – success! In fact, I learned some new words: splogging and flogging. I suppose that I won’t run across these regularly in my writing, but since they explain phenomena in the blogoverse, I thought that I would share them with you:

Splogging: is a term describing blog plagiarists. Sometimes this activity is also called blog scraping. The worst offenders take someone else’s content and re-post it and then earn advertising revenue based on visits to the stolen content. Of course there are degrees of scraping, but basically I have learned that it is naughty blogger behaviour. There are some companies/sites that have automated tools to do this – which expands the problem very quickly.

Flogging: an internet term that describes a fake blog, where a ghostwriter or marketing firm is hired to create content as a marketing tool and misrepresents the voice by pretending to be someone else. See The Consumerist’s article ‘Sony’s PSP blog flog revealed’.

In addition to learning a little bit more about blogging and podcasting from the seminar, I have started to become very interested in the implications of web 2.0 on science teaching, research and commercialization. I think this might become a recurring topic – once I find the time.