Scientists Who Fake Reviews of Their Papers

The pressure to publish is being blamed for the latest uptick in cases of scientific fraud and the appearance of new and devious means of getting one's work in print. An ongoing case involves a South Korean scientist who submitted glowing reviews of his own submitted papers (The Chronicle of Higher Education).

Hyung-In Moon submitted a manuscript to the Journal of Enzyme Inhibition and Medicinal Chemistry along with the names and email addresses of potential reviewers. So far, so good....many journals allow, or even require, such a list of reviewers by authors. The associate editor who handled the paper sent it out for review but became suspicious when they came back within 48 hours (those of us who have served as associate editors will recognize that the odds of two reviewers responding so quickly to be about the same as winning the recent Mega-Powerball lottery). The editor then sent the paper to two reviewers of his choosing; however, they recommended publication with some revisions, so the paper was published in 2010.

Apparently flush with success, Moon continued to submit manuscripts to JEIMC over the next year. According to an interview by the Chronicle, the associate editor who handled Moon's initial paper was still suspicious and took a closer look at the recommendations for reviewers. Although the referees appeared to be real scientists, the email addresses provided by Moon were commercial (g-mail, yahoo) rather than institutional. The editor finally wised up and contacted the reviewers directly. Not surprisingly, the email addresses provided were not theirs, and they had never heard of Moon.

Turns out, Moon had set up fake email addresses so that he could receive the journal's request for a review, and then it was a simple matter to send in a glowing review of his own work. When questioned, he apparently admitted this chicanery to the editor as well as to fabricating some of his data (!). The ensuing investigation has resulted in the retraction of his papers from various journals. According to Retraction Watch, 35 papers authored by Moon have been retracted.

Here is part of the retraction notice for the papers that were published in JEIMC:

"The peer-review process for all of the above articles was found to have been compromised and inappropriately influenced by the corresponding author, Professor HI Moon. As a result the findings and conclusions of these articles cannot be relied upon.

The corresponding author and the publisher wish to retract these papers to preserve the integrity of material published in the journal. The publisher acknowledges that the integrity of the peer review process should have been subject to more rigorous verification to ensure the reviews provided were genuine and impartial. The publisher apologizes for any inconvenience rendered to the readers of the journal and wishes to assure the reader that measures have been taken to ensure that the peer review process is comprehensively checked to avoid a similar error occurring."

Another journal editor-in-chief (Immunopharmacology and Immunotoxicology) has stepped down after 20 papers he handled by Moon were retracted.

To add insult to injury, Moon has suggested ways to avoid such problems in the future:

"There is nothing wrong with soliciting reviewers from authors, as long as there are some checks. Of course, authors will ask for their friends, but Editors are supposed to check they are not from the same institution or coauthors on previous papers. I know so many journals ask for *potential* reviewers, which they then add to a database of reviewers for the field the submission was made in. They then send the paper for review to other people on that database."  

He points out that it's the responsibility of the editor and the journal to check the feasibility of potential reviewers. It seems that he saw a loophole and took advantage of it....so we shouldn't be so hard on him. 

When I was an associate editor, I never used the reviewers suggested by the corresponding author unless I was pretty certain that the suggested reviewer had no conflict with the author (and in those cases always solicited another review from someone of my choosing). I could not imagine being comfortable accepting two reviews by people suggested by any author. It may take a bit more effort, but it's relatively easy to identify appropriate reviewers from the paper's list of citations, the journal's database, and an internet search of the paper's topic.

And accepting without question the names and addresses of suggested reviewers from an author is just plain foolish.

Nature Calling

The phone rings. You answer and it's a writer with Nature News who wants your opinion on an upcoming paper to be published in one of the Nature journals. You agree, and an embargoed copy is sent to you.

Upon reading the article, your first thought is, "How did this ever get past the reviewers?" You've never heard of the authors, who have never published anything of note on the topic.

Then you see that the authors have commented on an aspect (which they did not actually research) that is your area of expertise. Their statement goes something like this: "We know very little about Phenomenon X...due to the lack of research on the topic." They fail to cite your work or that of other researchers who have, in fact, published extensively on Phenomenon X.

Your response options:

a. Be gracious and compliment the authors for focusing attention on an important topic, endangered habitat/species, etc.

b. Be critical of the work and point out inaccuracies and false assumptions in the work that compromise the conclusions.

c. Ignore the findings of the paper and plug your own work.

d. Decline the request.

What would you do?

Whatever you do, you want to do it quickly if you wish your comments to be included in the news article. Journalists work on short deadlines, something that scientists often fail to appreciate. In this case, the news article is published by the journal that published the scientific paper. So they are likely not looking for extremely critical comments, which reflect badly on their decision to publish the paper. However, you want to provide an honest opinion--that's what they are asking for.

If you feel you cannot say anything positive about the paper, then option (d) is probably the best course of action. If you can say something positive, lead with this statement ("this work calls attention to a very important topic").  You can then say that there are still some aspects that need further research and/or that were not addressed in this paper. At this point, you may be able to mention your own work ("several researchers, including myself, are actively pursuing this aspect").

Another tip is to say something unusual or colorful. Such quotes are often used over more mundane ones. This all assumes, of course, that you wish to be quoted. There are advantages and disadvantages to being mentioned in a news article. You should carefully consider the consequences, depending on the situation. In this specific case I describe, the advantages usually outweigh any downside (commenter's profiles and links to their websites are usually included in Nature News articles).

Another option, not included in the choices above, is to write a rebuttal letter. This course of action is certainly one that would allow you to control what is printed as well as being able to fully expound on your criticism. I've never written a letter like this. However, I have written a "Reply" paper...one that was in response to a criticism of one of my papers (the journal published criticisms and replies in the form of short papers instead of letters). I described this experience in a previous blog post.

In general, however, I think there is a danger of critical letters to the editor sounding like "sour grapes". My reaction to a paper with which I disagree is to publish my own paper that addresses the points of contention.  It's a better use of your time and energy.

Don't Tell Anyone I Gave This To You

In a previous post, I described an ethical dilemma, one that probably occurs fairly frequently. I've encountered variations of it during my career.  A full description of this dilemma along with an expert opinion can be found here.  The following is my modified version that is similar to one of my experiences:

Cynthia is an ambitious post-doc having a problem with one of her laboratory techniques--extracting an enzyme from plant tissue containing lots of phenolic compounds (which bind proteins).  She's been trying for weeks to resolve this, but has been unsuccessful.  She is at her wit's end and finally goes to her PI to ask for help.  After listening to her tale of woe, he tells Cynthia not to worry--that he'll have a solution for her tomorrow.  The next day, she finds a manuscript on her desk with a note from the PI.  It says, "Check out the methods section...it has the solution to your problem.  However, don't make a copy of this or give it to anyone else.  Also, don't tell anyone that I gave this to you." She tries the method described in the paper and lo and behold, it works! When she excitedly reports this to her PI, she says, "I've scoured the literature and can find no mention of this technique.  Where did you get this paper?" The PI smiles mysteriously and says, "I've got dozens of them in my files."

I asked if the PI's actions were unethical.  The answer is, it depends.  Here is a summary of the expert's opinion (see the link above for the full version):

The moral dilemma hinges on the source of the paper.  Was it one of the PI's old, unpublished papers? One of his student's unpublished papers? Or was it a manuscript he got for review?

1. If the paper was written by the PI (and based on work done in his lab), then the PI is free to give the information and data to the post-doc to use.

2.  What may be less clear is why the former student's paper might also be given by the PI to the post-doc.  In this example, the student never published the paper and has left the university.  The post-doc can be given access to the contents because the PI's university likely owns the student's work as intellectual property.  According to the expert, the only way the student can claim the work is if s/he had previously gotten university permission to copyright the material.  Contrary to what many people think, the work conducted by a researcher (including their ideas) at an organization such as a university becomes the intellectual property of that organization.  In this case, the former student had no such copyright, so the PI, acting as the university's representative, has the authority to give the post-doc access to the contents of the paper.  It would be appropriate to contact the student to notify them that their unpublished information is to be used--and then cite them as the source in the acknowledgments section.  Unless the student makes a substantial contribution to the writing of the paper and a major intellectual contribution to the current work, then it is not appropriate to make them an author (the post-doc and the PI could potentially offer this option to the former student).

3. If the paper is one that the PI received for review, then giving it to the post-doc without the permission of the journal or the author is unethical.  The PI may be conflicted over his desire to help his stressed-out post-doc--and this may take precedence in his decision.  However, doing so is a breach of the confidentiality agreement that he entered into in accepting the role of reviewer.  In other words, the confidentiality agreement takes precedence over the PI's "moral" obligation to help the post-doc.  Furthermore, the PI is not exercising good judgment about how they will eventually use the information in their own publication and how to acknowledge the source of the information.  Giving the paper and the method to the post-doc may solve her immediate problem, but has created an even bigger problem for her in the future.  Unauthorized use of intellectual property obtained by privileged communication (review process) in another paper or proposal is plagiarism.  If they publish based on this method, they would be representing these ideas as their own, but which were taken from someone else's work.  If caught, both the PI and the post-doc could be subject to severe sanctions.

The expert opinion goes on to point out the likelihood that the plagiarism will be uncovered eventually.  If you think about it, any future paper by this PI and post-doc containing the plagiarized material will likely be read by the author of the original paper.  The author and the PI clearly work in the same specialized field, which is why the PI got the paper for review.  Isn't it just as likely that the author will get the PI's future paper for review?  In any case, the author will eventually see it when it is published.  This outcome is especially likely in a highly specialized field in which there are few experts.

My take on this scenario is that it is probably more common than you think.  I'm aware of colleagues who pass around papers they are reviewing or discuss the contents with others.  When confronted, they may admit they shouldn't do it, but then act as if it is nothing of great consequence.  Some pass on papers to students or post-docs (e.g., as exercises).  They may remove the author's name and affiliation, but the content of the manuscript is still confidential and should not be shown to anyone else or copied.  What if your student copies something from that paper without your knowledge, and it eventually ends up in a proposal or paper--where it is later recognized by the original author?

Sometimes you hear about reviewers asking journals for permission to have their post-doc or students review a manuscript (or maybe even do so without asking).  I don't think this is a good idea either.  If you don't have time to do a review, then decline and provide a list of people to substitute for you (you can suggest your post-doc); then, the journal can decide if your recommendation is a suitable reviewer, and the review will occur without your direct involvement.

The ethical situation described above and the hypothetical actions of the PI (#3) illustrate how easy it is for someone to get into deep trouble if they fail to take the time to consider the consequences of their actions.  I can easily imagine a PI who might make such a decision hastily and/or without thinking--but with no real malicious intent to injure the author.  However, such a decision commits at least two ethical transgressions--passing along confidential information without permission and putting another person into a tenuous and potentially liable situation (plagiarism would be the third, if they take the final step and publish).  The PI's moral obligation to "help" his post-doc clouds the larger ethical issues, and in the end, his action could instead seriously harm his post-doc's reputation and career.

Perhaps you have done something similar to this--we all make mistakes at some point, especially when we are inexperienced or under pressure.  Most people, though, would likely have a nagging feeling in their gut that such an action is wrong.  If you have this feeling about something you are facing or that someone else is telling you, pay attention.  Your gut is probably right.

Image Credit (modified from http://www.uwo.ca/nca/education/images/student_3.jpg)

Constructive Criticism III

This post continues my musings about criticism. We are now at the final stage of maturation in the development from novice to mature scientist in terms of accepting and providing criticism:

Late Career (full professor, senior scientist and beyond)

As I’ve tried to convey in the previous two posts, these transitions in one’s perception of and reaction to criticism accompany the change from a position in which one only receives criticism (e.g., a student) to someone who additionally serves as a critic (e.g., advisor, reviewer, editor). As a scientist’s experiences as a critic expand, her viewpoint regarding critiques of her own work begin to shift. At least, that’s the hope.

In addition to a better understanding of the critic’s viewpoint, the mature scientist is more informed about scientific standards in one’s field, expectations of reviewers and editors, and the reality of scientific “politics”. The mature scientist may develop a “thicker skin” to buffer against the vagaries of reviewer comments, particularly the harsher ones that border on (or are) personal attacks. This dermal protection may be enhanced by the recognition that vigorous criticism of one’s work sometimes indicates professional jealousy or a deep-seated inferiority complex on the part of the reviewer. As this week’s quote (side panel) suggests, it’s far better to be the subject of criticism than to be ignored. But that sentiment is small consolation when your work has been shredded by another.

It’s also useful to keep in mind that it’s the reviewer’s job to try to poke holes in your work. You actually want them to do this, because they will often identify weak points that you will then be able to address before publication. Some critics are a bit more enthusiastic in this regard than others. A few clearly have an ax to grind (either with you or what your work represents). Others are too lenient or lazy.

It goes without saying that it’s far better to receive constructive, rather than destructive, criticism. Even the most inexperienced person can recognize the difference—and the motives—behind these two types of criticism. Most people, upon receiving a critique, can readily see whether the comments are meant to help or to harm. Some reviews are a mixture of the two.

The goal for the author is to deal with each type of comment effectively. The mature author also knows that she cannot depend upon the editor to make the distinction between helpful and harmful comments (or act upon this knowledge). So she must learn to respond with finesse rather than with emotion. I realize this is far easier said, than done.

However, keep the goal in mind: getting your paper past the goal-keepers.

The mature author is not interested in “getting back at” the reviewers by ridiculing their remarks. Instead, she calmly explains where the reviewers may have gone wrong in their assessments (perhaps acknowledging that this may have been partly the author’s fault due to awkward wording) and then offers a well-worded explanation. She also honestly thanks the reviewers for being so thorough, so forthright, or whatever the case may be.

If the reviewer has written a mostly personal attack with no documentation to back up outrageous statements, then the mature author simply restates what her findings have shown and that in the absence of any contradictory evidence (which the reviewer has failed to provide), she has no alternative except to ignore the reviewer’s criticism (or provide only minimal response). By keeping her cool and refraining from outraged reactions to criticism, the mature scientist conveys a demeanor of professionalism and confidence. She doesn’t have to say that the reviewer is a jerk—it will be obvious.

In the end, the mature author derives much greater pleasure from out-maneuvering the mean-spirited critic than by reacting emotionally to a personal attack.

The mature scientist also spends considerable time critiquing others’ work--as advisor, reviewer, and/or editor. Giving constructive criticism is difficult to do well, however.  If you read articles about constructive criticism, they invariably focus on the technique of sandwiching the criticism between two compliments in an effort to disarm the recipient and avoid a defensive reaction. This method rarely works because it’s so transparent. Although a few skilled critics might pull it off, most people clumsily apply this technique, which then fails or backfires. This approach, which is akin to giving someone a bitter pill hidden in a piece of cake, is not how I define constructive criticism.

For criticism to be constructive (i.e., helping to improve), it must do more than simply compliment the author on some aspect of the work. The criticism should involve suggestions that will substantively improve the work. An example might be a suggestion to better display the results in a figure or table—an explicit description of how to do it, not just the statement that the change is needed. Or the suggestion might be a reorganization of information to better develop a logical argument—again, with explicit instructions as to how this might be accomplished. This all takes effort on the part of the critic, which is one reason it’s not often done well. Most critics will say that something needs improvement and then leave the author to figure out how to do it.

How much constructive criticism one gives is determined by a number of factors: time, motivation, etc. As I’ve discussed previously, it’s not always advisable to spend a lot of time on recommendations for revising a manuscript when the science is fatally flawed. If that’s the case, then it’s best to tell the authors that the study needs to be repeated or substantially augmented. The mature scientist resists the impulse to spend more time than necessary on such reviews. In other words, why waste time correcting flawed English, typos, and other editing problems when the manuscript is obviously not publishable? Copy editors will catch most, if not all, typos and other errors in an accepted manuscript. So the scientific reviewer is of most help in assessing the science and its significance. If you can provide one or two good suggestions for improvement, then you’ve likely helped the authors more than most reviewers.

Of course, not all authors (especially experienced ones) are grateful for explicit suggestions for improvement, so your efforts may be for naught. However, this should not deter you (in your role as constructive critic) from trying. I’ve found that in figuring out a way to improve someone else’s paper, I’ve added to my own toolbox.

Well, those are a few thoughts on constructive criticism in science, not by any means an exhaustive examination of the topic—and admittedly, a very personal view.  I’m sure others will have different viewpoints and experiences.

Constructive Criticism II

This post is the second in a series about criticism—how we evolve in our reactions to it and how we deal with it. Note that the suggested categories (and examples) are very general and not necessarily rigid. A post-doc might react to criticism as would someone more mature; conversely, a late-career scientist may still react to criticism as a student would.

I quizzed my spouse (who was always ahead of his peers in terms of maturity) about how he reacted to criticism as a student, and here is his response. He said that he automatically assumed that his advisor, other professors, even technical staff had many more years of experience than he did. He also believed that their criticism was always given to help him improve. Note that up to this point in his life, he had never encountered a truly nasty person and assumed the best in others (he still does, which actually works for him…maybe a topic for another post). Consequently, he invariably followed the suggestions of critics. He says he can’t recall any time that he strongly disagreed with any of them. I should point out that he was a very confident student…not at all a doormat. He simply was mature enough to accept criticism and smart enough to know that someone with 30+ years of experience might know a thing or two. I was in the same department, so I know his professors—all caring and honest people who tried hard to help students. Such is not likely the case everywhere, I realize.

My previous post focused on the less mature of us (me included) who progressed more or less in an average and predictable way. I also wanted to emphasize the negative outcomes of being either too sensitive to criticism or too resistant. Those who’ve matured early (like my husband) don’t need much help, in contrast to those who are resistant to criticism. Hence, my emphasis on cautionary tales.

So, kudos to those students and post-docs who have developed (early) a mature view of criticism. You are already at the next stage:

Mid-Career (asst. prof., new PI)

In this stage of maturation, the scientist has mostly moved beyond knee-jerk reactions to criticism, although there are occasional regressions, e.g., when a critic says something particularly bone-headed. During this phase, one listens carefully to the comments of others and distinguishes between the good, the bad, and the ugly.

Depending perhaps partly on personality and partly on early-career experiences, the mid-career scientist may begrudgingly accept criticism, but not like it very much. Others, particularly women, may still suffer self-doubts upon receiving criticism, whereas men may shrug off negative comments rather more quickly. A few begin to truly appreciate a thorough review that asks difficult questions and makes the author tighten up the logic or supply a better justification for a particular statement.

In terms of responses to criticism, the maturing author is diligent in responding to each and every critical comment—either making the suggested correction or calmly explaining why they disagree with the critic. This reaction is particularly important in responding to reviewer comments on a manuscript submitted to a journal. A good way to annoy a reviewer (who will be asked to do a second review of your paper) and the editor, is to fail to provide a thoughtful response to all previous comments and suggestions. A point-by-point reconciliation, explaining what you did in your revision and why, is greatly appreciated by reviewers and editors who must otherwise struggle to figure out whether you dealt effectively with their criticisms.

I know it’s sometimes difficult to remain detached when dealing with really idiotic comments that clearly indicate the reviewer either didn’t read your paper carefully or is simply out-to-lunch. But learning to become detached, taking the criticism less personally, and responding without undue emotion is what distinguishes the maturing scientist. At least on paper. It’s perfectly OK (in my opinion) to rail to your best friend, trusted colleague, or spouse about the awful review you just received and how unfair some of the comments were. You can even write out your immediate reactions to get it out of your system. But then you must step back, take some time away from it, and let yourself get some perspective on the critique. Perhaps first deal with the minor comments or those that you find useful. Let the ones that really get under your skin percolate a bit in your head (even if you are not thinking about them consciously, your subconscious is working on them). When you can contemplate the really negative comments without your blood pressure going up, then you are ready to address them.

Of course, it’s during this phase that you become a critic yourself, being tapped to carry out peer reviews and to serve on editorial boards. This experience accelerates the maturing process, as you might expect. You begin to see what kind of effort it takes to do a thorough review and to appreciate the time your colleagues have spent reviewing your work. You also see that associate editors have a difficult job (also unpaid) in persuading scientists to do reviews and to deal with author’s fragile egos.

What type of critic you become is likely influenced by how you have been treated as an author. You may decide to dish out the same negative stuff that you’ve received in the past, or conversely, commit to treating others with the respect and fairness you’ve failed to receive. Most scientists, fortunately, focus on providing constructive criticism and avoid personal attacks, either voluntarily or in response to journal admonishments to do so. A few become harsh reviewers, never finding anything of value in any manuscript that falls on their desks. Others are too easy, never finding any fault with any manuscript. You begin to realize that both these extremes are not helpful to you as an author.

In this mid-career stage, I think most scientists tend to accept all requests for reviews and then spend a lot of time on them, providing pages of comments. I have no statistics to back this up—just my impression from talking with others. My limited poll also suggests that women are more likely to fall into this behavioral category than men. I won’t speculate on why…. it’s probably for a lot of different reasons.

Personally, I never turned down a review request during this stage and felt obligated to put a great effort into doing a thorough and fair review. Doing a lot of reviews in the early to mid-career stages is profitable because you learn a great deal about writing, about good (and bad) science, about what journals want, and how to improve your own weaknesses. I provided some detailed reasons for being a reviewer at another blog (FemaleScienceProfessor), reproduced here:

1. You learn a lot about writing (e.g., in terms of technical proficiency and personal “style” or “voice”) and what it takes to get published. You learn as much from reviewing poor papers (what not to do) as from excellent papers (what distinguishes them from the pack).

2. You learn what the standards are (in general and for specific journals) and what specific criteria are used by reviewers and editors.

3. The more journals you review for, the better you understand what subject matter is being sought (and you are less likely to commit the error of submitting inappropriate material to a journal).

4. You become known to editors, especially if you are a consistently good reviewer. Editors then tend to go out of their way to handle future submissions of your own work personally and ensure that they are evaluated fairly and quickly.

5. As you become known as a fair and thorough reviewer to editors, you are in a better position to challenge a negative review of your own work

This has already become a longer post than I originally intended, so I’ll stop here and reserve further musings for the next post.

Time Saving Strategies (Part 3)

This is the third part of a series on time saving strategies: 1. Electronic communications (phone, email), 2. face-to-face communications (meetings, office drop-ins), and.... 

3. Other activities that require substantial time, but do not contribute to my scientific productivity.

  a. Scientific reviews.  Discussions with colleagues suggest that more and more of them are simply refusing to do any reviews for journals because they feel so overwhelmed, and the effort is not usually appreciated by superiors at performance review time. This is an extreme, and in my view inappropriate, reaction for professionals. We can't expect colleagues to review our papers if we do not reciprocate. A colleague told me the other day that when he inquired about a manuscript submitted many months earlier, he was told that they had asked and been turned down by 12 potential reviewers and were still looking for someone to accept the invitation to review.

I think if we dealt with reviews a bit more efficiently, however, we might be more inclined to do our share.

I found I was spending a lot of time doing reviews for journals.  I not only accepted too many assignments, but I spent too much time on unimportant aspects of the paper (editing) or tried to make suggestions for improvement when the work was fatally flawed. Some solutions I've implemented over the years:

     i. I decide how many reviews I will do per year and stick to this number. If I've published a paper in Journal X, then I will accept two review assignments from that journal during that year.  Requests from other journals are considered based on how much effort will be required, up to my quota for the year.  This number can be raised or lowered depending on other factors. Currently, I'm serving as assoc. editor for an international journal, which is taking a lot of time, so I'm turning down most requests from other journals.

     ii. I don't review a paper just because I'm interested in finding out what they did (i.e., it's very close to my area and I'm curious). If it's good, it'll get published and I'll see the information eventually. If it's not, then I didn't need to see it.

     iii. When I get a request for a review in my inbox, I don't respond immediately. Some editors send out requests to several people, and when two respond, the remainder are dis-invited.

     iv. When doing a review, I force myself to read the entire paper without stopping to make any comments or correcting any language problems.  I focus entirely on the science.  If there is a fatal flaw, then I focus in on that and don't expend any more time making suggestions for revision (because it's likely that the paper will be rejected and my time will be wasted).  If the work is solid, but the writing or some other aspect is weak, then I will make general comments and suggestions.  I do not spend time correcting typos or grammatical errors, which copy editors will handle later.  If the paper is excellent, then I will point out any minor edits that will improve understanding or presentation of the data.

  b. Administrative tasks. Filling out forms, doing performance reviews, etc.

     i. Automate as much as possible (save templates, set up automatic replies to frequent questions, prepare SOPs to guide routines of staff so that you don't have to repeat instructions)

    ii. Delegate (have others fill out forms or find information that takes time to locate, do purchasing and associated paperwork)

    iii. Make decisions quickly for tasks that are either not critical or can be reversed easily (in other words, don't spend hours agonizing over something that can be corrected later)

     iv. If you have a large staff, performance reviews may become time consuming or fall at a time when you have other important deadlines.  One way to decrease the time you spend evaluating subordinates is to have them write up a detailed summary of their accomplishments and progress on projects as well as plans and expectations for the next evaluation interval.  This approach lets the employee or student ensure that no accomplishment is overlooked and frees you from having to locate and collate all this information yourself.

For your own performance reviews, keep a running log of accomplishments by category or routinely add these to your CV.  Don't wait until right before your review (or when you start looking for a job) and then try to remember all of them.

      v. Write out directions for complicated procedures--ones that you spent a lot of time figuring out--so that you don't have to "reinvent the wheel" every time.  I have in mind here those annoying (non-science) tasks that happen so infrequently that they don't go into long-term memory (at least not mine).  All together, these procedures can add up to a lot of time wasted if you have to work them out from scratch each time.  I keep an electronic folder containing directions for various admin. or computer-based tasks and any shortcuts I've discovered.  You would be surprised at how many people fail to do this...even protest that they don't have time....

Well, that's all I can think of at the moment.  I'm sure some of you have pet peeves that consume your time and energy and that don't directly contribute to your science productivity.  Feel free to add to my list.

The "Black Box" of Peer Review

At the Inside Higher Ed, there is a review of a new book called How Professors Think: Inside the Curious World of Academic Judgment (go here to read the review).

The author claims to expose what goes on behind the closed doors of review panels, mostly for grants and fellowships. Not having read the book yet, I can't comment on its contents.

However, I have served on review panels for major funding agencies. Most panelists try to be fair, but there can be some bias on occasion (for/against colleagues, institutions, or certain topics). I know because I've seen it happen.

However, I make it a point to speak up promptly whenever I see bias creeping into the deliberations. In a recent instance, a panelist on a national review panel tried to push a proposal (written by a buddy, although there was no clear conflict of interest) up in the ranking. The proposal in question had been initially ranked low because it did not address the stated priorities of the funding program (whereas all the top-ranked proposals had managed to do so). This panelist argued that we should not hold this against an otherwise good proposal. He made a pretty convincing case.

When the program director (and the other panelists) seemed to become swayed by this panelist's argument, I spoke up. I agreed that the proposal in question was a good one based on other criteria, but failed in this key area. My view was that a high ranking for this proposal would not be fair to the other proposers who did restrict themselves to the stated priorities.

When the panel member tried to dismiss my concerns, I said a bit more emphatically that "it's not fair to change the rules in the middle of the game, and particularly without telling any of the participants. If word gets out about this, the reputation of this program will be damaged." ( I usually try to avoid using "sports analogies", but I sensed that the mostly male participants would more likely understand my point if I used one here).

Fortunately, the program director came to his/her senses and backed me up. Had I not been on the panel, I have no doubt that this undeserving proposal would have been funded, and one of the deserving proposals would have lost out.

Deciding how to react when faced with obvious bias or even outright unethical behavior is difficult--especially if it puts you at odds with someone in authority. But failing to do something about it is far worse.

I guess my point is that we can bemoan the problems associated with the review process, but it's really up to us, individual scientists, to ensure that it's a fair process. Think about this the next time you are asked to serve on a panel or act as a reviewer of a proposal or manuscript. These are opportunities to influence the review system--and science--for the better. You might also learn how to write a better, more effective proposal or paper by seeing what constitutes an excellent, successful one from one that is rejected. More about this later....