The "Black Box" of Peer Review

At the Inside Higher Ed, there is a review of a new book called How Professors Think: Inside the Curious World of Academic Judgment (go here to read the review).

The author claims to expose what goes on behind the closed doors of review panels, mostly for grants and fellowships. Not having read the book yet, I can't comment on its contents.

However, I have served on review panels for major funding agencies. Most panelists try to be fair, but there can be some bias on occasion (for/against colleagues, institutions, or certain topics). I know because I've seen it happen.

However, I make it a point to speak up promptly whenever I see bias creeping into the deliberations. In a recent instance, a panelist on a national review panel tried to push a proposal (written by a buddy, although there was no clear conflict of interest) up in the ranking. The proposal in question had been initially ranked low because it did not address the stated priorities of the funding program (whereas all the top-ranked proposals had managed to do so). This panelist argued that we should not hold this against an otherwise good proposal. He made a pretty convincing case.

When the program director (and the other panelists) seemed to become swayed by this panelist's argument, I spoke up. I agreed that the proposal in question was a good one based on other criteria, but failed in this key area. My view was that a high ranking for this proposal would not be fair to the other proposers who did restrict themselves to the stated priorities.

When the panel member tried to dismiss my concerns, I said a bit more emphatically that "it's not fair to change the rules in the middle of the game, and particularly without telling any of the participants. If word gets out about this, the reputation of this program will be damaged." ( I usually try to avoid using "sports analogies", but I sensed that the mostly male participants would more likely understand my point if I used one here).

Fortunately, the program director came to his/her senses and backed me up. Had I not been on the panel, I have no doubt that this undeserving proposal would have been funded, and one of the deserving proposals would have lost out.

Deciding how to react when faced with obvious bias or even outright unethical behavior is difficult--especially if it puts you at odds with someone in authority. But failing to do something about it is far worse.

I guess my point is that we can bemoan the problems associated with the review process, but it's really up to us, individual scientists, to ensure that it's a fair process. Think about this the next time you are asked to serve on a panel or act as a reviewer of a proposal or manuscript. These are opportunities to influence the review system--and science--for the better. You might also learn how to write a better, more effective proposal or paper by seeing what constitutes an excellent, successful one from one that is rejected. More about this later....