Can You Write Me a Reference Letter, Preferably a Good One?

You've carefully put together your application package for that internship, fellowship, or dream job, including a compelling statement about your research or teaching goals and a list of all of your publications, awards, grants, and other accomplishments. You think you are well qualified, with achievements that are above average for someone at your career stage. You submit your application and then wait anxiously for that email telling you that you've been invited for an interview or selected for that fellowship or internship.

When it finally comes, the news is not good: "Dear Candidate, we regret to inform you that your application was not successful. There were many excellent candidates this year but....". At the end is a personal note that says, "You might want to rethink your list of references."

Huh? What could that mean? You are not sure, but it sounds as if one of your reference letters was not exactly glowing or perhaps even negative. You think about the three people you asked to write a letter of support for you. They all were people who you thought would write an excellent recommendation. They all agreed to prepare a letter for you. What could have gone wrong and was this what sunk your application?

Reference letters have been on my mind recently since I just participated as a panelist for a student award. I was struck by the huge discrepancy among reference letters, even those written for the same student. Here are some excerpts (modified slightly to hide any identifying information) followed by my assessment:

"I strongly recommend Ms. X for this award. She is hard-working, eager to learn, and quite intuitive. Her academic record is outstanding with a 4.0 grade-point average. She routinely asks penetrating questions in class and shows a more in-depth grasp of the topic than most students. She has developed a unique angle for her thesis research project, which puts it at the cutting edge of the field. Among the 100 or so graduate students I have interacted with as professor, I would rank Ms. X in the top 5%." The letter goes on for several paragraphs to extol the many virtues of this student, providing detailed examples and heart-felt opinions. My opinion: there is no doubt in my mind that this student is held in high esteem by her professor and deserves serious consideration.

"I wish to recommend Ms. X for this award. She is a student in our department and has taken a couple of classes with me. I understand that her advisor is also writing a letter for her. She is conducting thesis research in an area related to mine, which is ...." This letter then goes on to describe the author's area of research, rather than the student's. This professor states no opinion about her abilities or skills or qualifications for this particular award. The letter ends with, "If you need any more information, please don't hesitate to contact me." My opinion: this person has no idea how to write a letter of reference and/or did not want to (or have time to) write a positive, substantive letter for this student (but agreed to write one anyway). The student made a serious mistake asking this professor for a letter.

"I am writing this reference at the request of Mr. Y who is applying for this award. I have been acquainted with this student for a short time. He is about to complete his second semester in our department and I believe he has just initiated his field research. Based on his undergraduate record and the accomplishments he lists in his CV, I can recommend him to receive this award. Feel free to contact me for further information." My opinion: this person knows next to nothing about this student and should not have been asked to write a letter.

"I am writing to offer my enthusiastic support for Mr. Z's application for this award. He has made excellent progress on his thesis research and is excelling in all his coursework. He has all the personal qualities of the most successful graduate student, and I expect him to make important contributions to our understanding of [research topic]. Mr. Z is a bright, motivated, and energetic young scientist that I am pleased to have in my lab. He is a worthy candidate for this award, and I strongly encourage you to consider him. I would be happy to provide additional details about Mr Z's qualities should you need more information. I unreservedly recommend Mr. Z to you for this award." The letter goes into great detail about the student's research and his role in its design and novel ideas for experiments. My opinion: this is a student whose professor has no reservations about recommending him for the award and deserves serious consideration.

The above examples should give you some idea of how varied real reference letters can be. In no case was there a letter that stated anything negative or described any perceived weaknesses. Yet it was easy to categorize letters as outstanding, good, fair, and poor. At the one extreme were the letters that contained glowing, but detailed, comments and that stated flat-out their unreserved recommendation of the student. At the other extreme were the shortest letters that contained no positive comments or even a single opinion about the student.

Some people were clearly outstanding writers of reference letters and understood the importance of providing detailed information and personal opinions about a candidate. Perhaps ten percent of the letters fell into a category I would call "glowing". In several cases, we got letters like number 3 above and even a couple that were from the student's major advisor, who had had them in their lab for five or six years. Yikes. In those instances, it was sometimes clear that the advisor did not hold the student in high regard but still agreed to write a letter. In one case, the letter from the (very busy, very famous) advisor was three sentences long and basically said "I recommend this person for this award". In other cases, it seemed that the letter-writer had no idea that their letter, which lacked any observations or opinions, positive or otherwise, would be perceived as a bad reference.

In the cases where the student got a glowing letter from one person and a poor letter from someone else (like Ms. X above), it was possible that the second letter-writer had some beef with either the student or the advisor. Even so, we could not discount it because at a minimum it showed that the student had not done their homework in selecting people to write a letter for them. Also, their overall rating would suffer in comparison with another student whose letters were all outstanding.

So what can you do, if anything, to ensure that your reference letters are all good ones? I'll explore that in upcoming posts.

Image credit: modified photo from morguefile.com

Can Movies Provide a Model for Assigning Scientific Credit?

Imagine if the credit for a scientific publication was ascribed the same way movie credits are.

What got me thinking about this was a reader comment on a paper published in PLoS Biology about author sequence and how to calculate credit for author contributions in multiauthor publications.

Determining author contributions to a scientific paper is an increasingly important topic as the number of multiauthor papers has increased over time. The reason this issue is of concern is because evaluation committees may use author position on a person's publications (along with other measures) as a criterion to gauge their scientific worth and leadership abilities. The commenter suggested that the solution (to the dilemma of calculating credit) might be resolved by using the movie credit model. A post on the PLoS blog also makes the argument for a movie credit-style approach:

"Such a research credit system would have huge benefits for one’s career prospects; and it might encourage more effective collaborations. Moreover, these credits could easily be tracked by scientist or project in a database akin to the Internet Movie Database (IMDB). It could provide an alternative to the ever-so-important citation factors as a means of assessing one’s scientific impact. And maybe one day there will even be an Academy Awards of Science."

Those of you who read this blog know I like movies and often use them to discuss, among other things, how women in science are perceived by society. So, of course, I found this suggestion intriguing and decided to delve a bit further. It's not as crazy as it sounds, so bear with me while I explore this idea a bit.

In a movie, there are producers, actors, set designers, cinematographers, screenwriters, and so forth. The credits that roll during the closing sequence list everyone who contributed to the making of the movie. Some contribute more than others and are listed first in the credits, but everyone is listed, from the director down to the lowliest person who held the lead actor's cigarette during takes. Relevant to our discussion is the fact that a movie is the end result of a technological effort involving a team of people making different contributions to the end product, very similar to a modern scientific paper. The movie, however, is never solely credited to a single person or category of filmmaker professional. The success of the movie, which depends on all of these people performing their jobs well, clearly cannot be ascribed to a single person or group of people such as the screenwriters (although some, such as the director, make a more substantial contribution than others). The Oscars and other awards, which are given to honor not only screen actors but also cinematographers, costume designers, and others, reflect the fact that filmmaking is a joint effort of a variety of film professionals.

Unlike other publications that are the result of the sole efforts of a single author (e.g., novels), modern scientific publications often reflect the work of a team of scientists, technicians, statisticians, graphic artists, and others in much the same way a movie is made. So why do all the people working on a movie get credit (which they claim on their resumes), while not everyone who contributes to a scientific paper can claim credit for it?

Interestingly, early movies (before 1960) listed only the director and actors (and maybe the costume designer if it was a period movie), and these came at the beginning of the film. Someone who worked as an illustrator would never get credit; only the art director might be given onscreen credit. The change to listing everyone who had anything to do with the making of a film was due to trade unions. The unions argued that their members should get credit because it was essential for getting jobs on future films and because non-actor artists should be able to list the movie on their resume. Nowadays, virtually everyone who is in filmmaking belongs to a trade union, which looks out for its members....hence, the 15 minute long credit sequences we see at the end of movies.

In scientific publications, the authors are really the only individuals who receive credit for the work and who can list the paper on their resume, even though a number of non-authors may have contributed to the final product, and in some cases, may have made a critical contribution (for example, a statistician who helped with the experimental design). Such people usually are mentioned only in the acknowledgements, where their contributions are lumped all together in what is essentially a footnote. Because of this, non-authors cannot list the publication on their resume....and can describe only their general role ("In 2012, I assisted in field sampling during a study of coral reefs in Indonesia"). In contrast, a cinematographer, an animator, a sound technician, or even a gaffer can list on their resume the movies they've worked on.

The assignment of credit for a scientific work in part reflects the nature and history of scientific research in which the early discoverer of new knowledge, who was the sole author of the paper, got the credit. This pattern held until the early 1900s when more scientists began collaborating and tackling complex projects requiring teams of people. The numbers of authors on a paper increased while each author's contribution became less clear and quantifiable. Scientific studies increasingly involved technicians and others whose contributions were considered insufficient to count as authors. The tradition of all credit going to part of the scientific team via authorship has persisted despite the increase in non-author contributors to scientific research. Some perhaps suggested the idea for the study; others provided the laboratory space or other essential logistics; many collected data or provided technological expertise; a few gave statistical guidance....none enough to warrant authorship or credit for a scientific discovery but who played a role in the ultimate outcome.

Many scientific publications today are produced by multi-disciplinary teams, each with many people contributing in much the same way as teams of animators or set designers on a film. Papers in some disciplines such as genomics can list 100 authors, a situation that is likely to increase in the future. There is even one recent paper describing the Large Hadron Collider that lists 2,926 authors. In such cases, how do you know who did what and who, other than the first author, made a major contribution? Although some journals (e.g., Nature journals) require an author contributions statement, these are often awkward and potentially lengthy. Would it be more feasible to use a movie credit model in such cases or would that lead to even more confusion?  

Could the movie credit model even work for scientific publications? Well, I can see some problems with it, not the least of which would be resistance from scientists accustomed to getting all the credit. At issue would be who should get credit for the scientific discovery reported in the paper and how would this be indicated in the credits? Of concern, also, might be that someone listed in the credits, but with a minor contribution, would claim a greater role on their CV. I don't think the latter would be any more of an issue than it is in the movie business (a lighting technician is not going to claim she made the movie). In fact, it might reduce this possibility. With the current model of listing multiple authors (using an often unspecified method of assigning position*), minor contributors can get undue credit.

Another stumbling block might be how to sequence the credits to indicate relative contribution. The credits could be listed in a sequence of decreasing contribution, much like they are today in author lists, except that their role would be clear (Principal Investigator, co-Principal Investigator, Laboratory Director, Field/LaboratoryTechnician, Statistician, etc.), and people who often are inappropriately listed as "authors" today would be more accurately credited for their actual role in the study. The Principal Investigator(s) (who conceived and designed the study, conducted the research, and wrote the paper) would be appropriately credited as the discoverer(s) of new knowledge. A movie credit-style listing might add to the length of a paper, but concern about length is likely to be less of an issue as we move to primarily electronic publication of scientific papers and away from printed journals.

The biggest obstacle, perhaps, would be coming up with a standardized system that everyone agrees upon and then how to get everyone to adopt it. It would likely take some of the major journals making the move first and showing how it might work. Several have already implemented the requirement for an author contributions statement, which is an attempt to go in this direction.

I'm sure there would be other objections to the movie credit model, and one could argue that it would not resolve the issue of how to quantify credit for a scientific paper. However, spelling out the roles that each person played might make it easier to quantify credit than using some formula based on the author's position in the author list (as proposed by the paper in PLoS Biology). The other thing such a change would do, which seems to be an improvement over the current model, is to allow those contributors who are now mentioned only in the acknowledgements to claim credit for their contribution to the work.

I'm not necessarily advocating a change from the traditional model to the movie credit model. Possibly, such a change would introduce unforeseen problems and perhaps add to the bickering that already occurs in some research groups as to author sequence and who should get credit for a discovery. However, it's an interesting exercise to contemplate how it might work and who might benefit.

The problem of how to quantify scientific credit is not going to go away and will only worsen as the average number of authors per paper increases. We eventually may be forced to consider a radically different credit system that better reflects the complexity of modern scientific research.

*Someone unfamiliar with how authorship position on scientific papers is determined might assume that the lower the position in the list, the lower the contribution and, thus, the less credit deserved. However, that logic does not always match how author position was determined, which can vary from discipline to discipline and even from lab to lab. Although the first author position is typically occupied by the person who deserves most of the credit (but not always), the ranking of the succeeding authors may have been determined alphabetically, for example, when there are collaborating groups. In some cultures, the last author position frequently goes to the person who deserves substantial credit; this is typically the PI who got the funding and who designed and oversaw the project. In others, the last position may go to the person who contributed the least. Consequently, some may benefit unduly from their position while others may be underrated, depending on how the author sequence is assessed relative to how it was actually determined. 

Image Credit: modified image from morguefile.com

Procrastination

Well, so much for my New Year’s resolution to be more productive. Maybe it has something to do with the fact that I’m traveling in a country with spectacular natural scenery and lots of interesting things to do?

Scientists often do sabbaticals and fieldwork in locations far from home where many more (and new) distractions exist than at home. How do you manage the work you are supposed to be doing and the opportunity to experience a new and interesting place?

Such a situation is problematic for the procrastinator and even for people like me who rarely put off important tasks:  “I’ll just take a quick hike to this overlook–shouldn’t take more than an hour round-trip–then I’ll feel invigorated enough to finish formatting that paper.” Hours later, after reaching the summit, which was considerably more challenging than it looked on the map, I’ve had to stop and rest and have something to eat before starting back down. By the time I get back, it’s time to start dinner. By 9 pm, I’ve finished cleaning up the kitchen but can’t keep my eyes open. Well, tomorrow is soon enough to tackle that formatting….”

At home, the above scenario would never happen to me. I’m good at ignoring distractions and prioritizing tasks. However, when I am traveling and encountering unique and never-before-seen places, sights that I may never see again in my lifetime, the decision to postpone working on a paper that only a handful of people will likely read becomes more reasonable. If you have only one or two days in a particular location before moving on and there are interesting things to see and do there, then you have to make a choice.

In the long-run, it makes more sense to take the time for new experiences….as long as you don’t avoid work altogether or miss important deadlines. 

Actually, I’ve been quite good at combining travel with work. Check and answer email and write or analyze data for a few hours in the morning. In the afternoon, go for a drive to check out an interesting local attraction. Come back and do some more work and answer any new emails.

Although my current procrastination is situation-specific, many people have a more generalized procrastination problem. That is, they routinely put off important tasks and never meet deadlines because they find a multitude of other, minor tasks (without deadlines) to do instead.

Some people are what are known as “productive procrastinators”, a term coined by Piers Steel, a professor who has published a book called, “The Procrastination Equation”. Some of the techniques he says he uses to productively procrastinate sound very familiar. For example, when I have multiple projects, I can procrastinate on one by working on another. Or when writing, I might put off penning the introduction by writing the methods or formatting the literature cited. When I’m working on a paper and find myself pondering whether it’s time to clean my desk, I will switch to another part of the paper, e.g., prepare a figure or table, and then go back later to the narrative after the impulse to procrastinate passes. Psychologically, this works because we naturally will work on something boring and repetitive if it allows us to avoid something even more distasteful or daunting.

Much earlier, Robert Benchley revealed his secret to accomplishing a lot of work through productive procrastination: “anyone can do any amount of work, provided it isn’t the work he is supposed to be doing at that moment.” I’ve used this principle for years. I might spend a few hours during the workday cleaning up my desktop, perusing a scientific catalogue, or just thinking about new project, but in the evening or on the weekend will write or analyze data or proofread a manuscript. In other words, I’m not doing what I’m supposed to be doing during normal work hours but doing them instead when I’m off the clock. This strategy is a psychological “trick” that allows you to work on whatever you feel like tackling at a particular time but which leads to an overall net production.

Let me hasten to add that the total time I spend actually working on stuff I’m supposed to be working on (research, data analysis, writing) has traditionally exceeded 40 hours per week. And over my career, I’ve been reasonably productive, publishing more than 100 peer-reviewed papers and book chapters. If you are goofing off most of the time, surfing the internet, reading blogs like this one, and only spending a fraction of the day actually producing something substantive, then you are going to have a problem. 

If you want to find out where you rank relative to others, you can take the procrastination test. 

Scientists Who Fake Reviews of Their Papers

The pressure to publish is being blamed for the latest uptick in cases of scientific fraud and the appearance of new and devious means of getting one's work in print. An ongoing case involves a South Korean scientist who submitted glowing reviews of his own submitted papers (The Chronicle of Higher Education).

Hyung-In Moon submitted a manuscript to the Journal of Enzyme Inhibition and Medicinal Chemistry along with the names and email addresses of potential reviewers. So far, so good....many journals allow, or even require, such a list of reviewers by authors. The associate editor who handled the paper sent it out for review but became suspicious when they came back within 48 hours (those of us who have served as associate editors will recognize that the odds of two reviewers responding so quickly to be about the same as winning the recent Mega-Powerball lottery). The editor then sent the paper to two reviewers of his choosing; however, they recommended publication with some revisions, so the paper was published in 2010.

Apparently flush with success, Moon continued to submit manuscripts to JEIMC over the next year. According to an interview by the Chronicle, the associate editor who handled Moon's initial paper was still suspicious and took a closer look at the recommendations for reviewers. Although the referees appeared to be real scientists, the email addresses provided by Moon were commercial (g-mail, yahoo) rather than institutional. The editor finally wised up and contacted the reviewers directly. Not surprisingly, the email addresses provided were not theirs, and they had never heard of Moon.

Turns out, Moon had set up fake email addresses so that he could receive the journal's request for a review, and then it was a simple matter to send in a glowing review of his own work. When questioned, he apparently admitted this chicanery to the editor as well as to fabricating some of his data (!). The ensuing investigation has resulted in the retraction of his papers from various journals. According to Retraction Watch, 35 papers authored by Moon have been retracted.

Here is part of the retraction notice for the papers that were published in JEIMC:

"The peer-review process for all of the above articles was found to have been compromised and inappropriately influenced by the corresponding author, Professor HI Moon. As a result the findings and conclusions of these articles cannot be relied upon.

The corresponding author and the publisher wish to retract these papers to preserve the integrity of material published in the journal. The publisher acknowledges that the integrity of the peer review process should have been subject to more rigorous verification to ensure the reviews provided were genuine and impartial. The publisher apologizes for any inconvenience rendered to the readers of the journal and wishes to assure the reader that measures have been taken to ensure that the peer review process is comprehensively checked to avoid a similar error occurring."

Another journal editor-in-chief (Immunopharmacology and Immunotoxicology) has stepped down after 20 papers he handled by Moon were retracted.

To add insult to injury, Moon has suggested ways to avoid such problems in the future:

"There is nothing wrong with soliciting reviewers from authors, as long as there are some checks. Of course, authors will ask for their friends, but Editors are supposed to check they are not from the same institution or coauthors on previous papers. I know so many journals ask for *potential* reviewers, which they then add to a database of reviewers for the field the submission was made in. They then send the paper for review to other people on that database."  

He points out that it's the responsibility of the editor and the journal to check the feasibility of potential reviewers. It seems that he saw a loophole and took advantage of it....so we shouldn't be so hard on him. 

When I was an associate editor, I never used the reviewers suggested by the corresponding author unless I was pretty certain that the suggested reviewer had no conflict with the author (and in those cases always solicited another review from someone of my choosing). I could not imagine being comfortable accepting two reviews by people suggested by any author. It may take a bit more effort, but it's relatively easy to identify appropriate reviewers from the paper's list of citations, the journal's database, and an internet search of the paper's topic.

And accepting without question the names and addresses of suggested reviewers from an author is just plain foolish.

New Year's Resolution: Be More Productive

Looking for advice on how to be more productive in the coming year? Check out this entertaining video about "The Science of Productivity", created by ASAPScience. In it, the creators discuss something called "deliberate practice", which we've discussed here previously.

Single-Space Me

Do you type one or two spaces after a period in a sentence? What were you taught?

I was taught to use two spaces, but I never realized that type setters are adamant about the use of one space between sentences. After forty years of writing, I find it extremely difficult not to double-space after a period. As I write this post, I am having difficulty not putting two spaces after typing that period.  See? I did it again.

Anyway, check out this article for a history of the two vs. one space rule and the rationale for the latter.

More Gender Bias Uncovered: Conference Symposia Organizers

Think about the last scientific conference you attended.  Did the speaker roster reflect the composition of your science society?  At a recent meeting I attended, several of us (women) were discussing the fact that there seemed to be a preponderance of male speakers, a proportion that did not reflect the gender makeup of our society.  When we pointed this out to the (mostly male) organizers, they basically pooh-poohed our concerns and waved their arms at all the female participants milling around (mostly grad students)....as if that addressed the point we were trying to make.

Turns out, there is now some evidence that the gender of session organizers has a significant influence on who ends up speaking in the more prestigious symposia sessions.

A recently published paper in PLOS One, "Stag Parties Linger: Continued Gender Bias in a Female-Rich Scientific Discipline", examined participation by women and men at conferences in a field in which women are in the numerical majority.  Based on an analysis of 21 annual meetings of the American Association of Physical Anthropologists, they found that in the subfield of primatology:

1. Women gave more posters than talks; men gave more talks than posters.

2. The proportion of female participants was lower in symposia organized by men (29%) compared to those organized by women only (64%) or by both men and women (58%).

They found the same pattern for 12 annual meetings of the American Society of Primatologists.

Primatology is a field that has had more female than male participants since the early 1970s and where women have achieved substantial peer and popular recognition for their achievements (Jane Goodall, Dian Fossey).  The results of this study suggest that even numerical dominance by females in a scientific field does not prevent gender bias. By looking at symposia participation, which is typically by invitation of the organizer, in relation to the gender of the organizer(s), the authors were able to say something about whether the underrepresentation by women at conferences might be due to lack of assertiveness by females or to bias.  Whereas posters and regular talks are self-motivated, the more prestigious symposia reflect a choice made by the organizer.  The results suggest that when men are the organizers, fewer women are invited to participate, but that men and women are equal participants in symposia organized by women (despite the larger numbers of women in the field).

Such bias may be unconscious on the part of the male symposia organizers.  I'm just guessing that this is the case as the authors have no data either way.  The authors of this paper, however, do state that they "cannot rule out" the possibility that male primatologists are more homophilic (homophily is preferential interaction with others who have similar attitudes, beliefs, or personal characteristics) than women.  The idea is that male symposia organizers have to spend more time interacting with their invitees, both before and after the symposium (e.g., to jointly prepare a conference proceedings) and simply prefer to work with other males.  Perhaps.  The jury is still out on that one.  But my guess is that this behavior is not restricted to male primatologists.

The finding that women gave more posters than talks and men the reverse is troubling and suggests the possibility that women are preferring the less visible presentation mode.  Although the study indicated that men were 12% more likely to request an oral talk over poster than women, which might explain the gender difference, the authors suggest that there still might be some bias occurring at the presentation selection/assignment stage (by the conference organizers).  Their data were pretty weak on this point, however.

In any case, this is yet another example of why it is important for women to be assertive in promoting themselves and their science.  We may need to encourage female students to give oral presentations. As for bias in inviting speakers, one solution is for more women to organize conference symposia and ensure that women are well represented in the speaker lineup.