Can You Write Me a Reference Letter, Preferably a Good One?

You've carefully put together your application package for that internship, fellowship, or dream job, including a compelling statement about your research or teaching goals and a list of all of your publications, awards, grants, and other accomplishments. You think you are well qualified, with achievements that are above average for someone at your career stage. You submit your application and then wait anxiously for that email telling you that you've been invited for an interview or selected for that fellowship or internship.

When it finally comes, the news is not good: "Dear Candidate, we regret to inform you that your application was not successful. There were many excellent candidates this year but....". At the end is a personal note that says, "You might want to rethink your list of references."

Huh? What could that mean? You are not sure, but it sounds as if one of your reference letters was not exactly glowing or perhaps even negative. You think about the three people you asked to write a letter of support for you. They all were people who you thought would write an excellent recommendation. They all agreed to prepare a letter for you. What could have gone wrong and was this what sunk your application?

Reference letters have been on my mind recently since I just participated as a panelist for a student award. I was struck by the huge discrepancy among reference letters, even those written for the same student. Here are some excerpts (modified slightly to hide any identifying information) followed by my assessment:

"I strongly recommend Ms. X for this award. She is hard-working, eager to learn, and quite intuitive. Her academic record is outstanding with a 4.0 grade-point average. She routinely asks penetrating questions in class and shows a more in-depth grasp of the topic than most students. She has developed a unique angle for her thesis research project, which puts it at the cutting edge of the field. Among the 100 or so graduate students I have interacted with as professor, I would rank Ms. X in the top 5%." The letter goes on for several paragraphs to extol the many virtues of this student, providing detailed examples and heart-felt opinions. My opinion: there is no doubt in my mind that this student is held in high esteem by her professor and deserves serious consideration.

"I wish to recommend Ms. X for this award. She is a student in our department and has taken a couple of classes with me. I understand that her advisor is also writing a letter for her. She is conducting thesis research in an area related to mine, which is ...." This letter then goes on to describe the author's area of research, rather than the student's. This professor states no opinion about her abilities or skills or qualifications for this particular award. The letter ends with, "If you need any more information, please don't hesitate to contact me." My opinion: this person has no idea how to write a letter of reference and/or did not want to (or have time to) write a positive, substantive letter for this student (but agreed to write one anyway). The student made a serious mistake asking this professor for a letter.

"I am writing this reference at the request of Mr. Y who is applying for this award. I have been acquainted with this student for a short time. He is about to complete his second semester in our department and I believe he has just initiated his field research. Based on his undergraduate record and the accomplishments he lists in his CV, I can recommend him to receive this award. Feel free to contact me for further information." My opinion: this person knows next to nothing about this student and should not have been asked to write a letter.

"I am writing to offer my enthusiastic support for Mr. Z's application for this award. He has made excellent progress on his thesis research and is excelling in all his coursework. He has all the personal qualities of the most successful graduate student, and I expect him to make important contributions to our understanding of [research topic]. Mr. Z is a bright, motivated, and energetic young scientist that I am pleased to have in my lab. He is a worthy candidate for this award, and I strongly encourage you to consider him. I would be happy to provide additional details about Mr Z's qualities should you need more information. I unreservedly recommend Mr. Z to you for this award." The letter goes into great detail about the student's research and his role in its design and novel ideas for experiments. My opinion: this is a student whose professor has no reservations about recommending him for the award and deserves serious consideration.

The above examples should give you some idea of how varied real reference letters can be. In no case was there a letter that stated anything negative or described any perceived weaknesses. Yet it was easy to categorize letters as outstanding, good, fair, and poor. At the one extreme were the letters that contained glowing, but detailed, comments and that stated flat-out their unreserved recommendation of the student. At the other extreme were the shortest letters that contained no positive comments or even a single opinion about the student.

Some people were clearly outstanding writers of reference letters and understood the importance of providing detailed information and personal opinions about a candidate. Perhaps ten percent of the letters fell into a category I would call "glowing". In several cases, we got letters like number 3 above and even a couple that were from the student's major advisor, who had had them in their lab for five or six years. Yikes. In those instances, it was sometimes clear that the advisor did not hold the student in high regard but still agreed to write a letter. In one case, the letter from the (very busy, very famous) advisor was three sentences long and basically said "I recommend this person for this award". In other cases, it seemed that the letter-writer had no idea that their letter, which lacked any observations or opinions, positive or otherwise, would be perceived as a bad reference.

In the cases where the student got a glowing letter from one person and a poor letter from someone else (like Ms. X above), it was possible that the second letter-writer had some beef with either the student or the advisor. Even so, we could not discount it because at a minimum it showed that the student had not done their homework in selecting people to write a letter for them. Also, their overall rating would suffer in comparison with another student whose letters were all outstanding.

So what can you do, if anything, to ensure that your reference letters are all good ones? I'll explore that in upcoming posts.

Image credit: modified photo from morguefile.com

Can Movies Provide a Model for Assigning Scientific Credit?

Imagine if the credit for a scientific publication was ascribed the same way movie credits are.

What got me thinking about this was a reader comment on a paper published in PLoS Biology about author sequence and how to calculate credit for author contributions in multiauthor publications.

Determining author contributions to a scientific paper is an increasingly important topic as the number of multiauthor papers has increased over time. The reason this issue is of concern is because evaluation committees may use author position on a person's publications (along with other measures) as a criterion to gauge their scientific worth and leadership abilities. The commenter suggested that the solution (to the dilemma of calculating credit) might be resolved by using the movie credit model. A post on the PLoS blog also makes the argument for a movie credit-style approach:

"Such a research credit system would have huge benefits for one’s career prospects; and it might encourage more effective collaborations. Moreover, these credits could easily be tracked by scientist or project in a database akin to the Internet Movie Database (IMDB). It could provide an alternative to the ever-so-important citation factors as a means of assessing one’s scientific impact. And maybe one day there will even be an Academy Awards of Science."

Those of you who read this blog know I like movies and often use them to discuss, among other things, how women in science are perceived by society. So, of course, I found this suggestion intriguing and decided to delve a bit further. It's not as crazy as it sounds, so bear with me while I explore this idea a bit.

In a movie, there are producers, actors, set designers, cinematographers, screenwriters, and so forth. The credits that roll during the closing sequence list everyone who contributed to the making of the movie. Some contribute more than others and are listed first in the credits, but everyone is listed, from the director down to the lowliest person who held the lead actor's cigarette during takes. Relevant to our discussion is the fact that a movie is the end result of a technological effort involving a team of people making different contributions to the end product, very similar to a modern scientific paper. The movie, however, is never solely credited to a single person or category of filmmaker professional. The success of the movie, which depends on all of these people performing their jobs well, clearly cannot be ascribed to a single person or group of people such as the screenwriters (although some, such as the director, make a more substantial contribution than others). The Oscars and other awards, which are given to honor not only screen actors but also cinematographers, costume designers, and others, reflect the fact that filmmaking is a joint effort of a variety of film professionals.

Unlike other publications that are the result of the sole efforts of a single author (e.g., novels), modern scientific publications often reflect the work of a team of scientists, technicians, statisticians, graphic artists, and others in much the same way a movie is made. So why do all the people working on a movie get credit (which they claim on their resumes), while not everyone who contributes to a scientific paper can claim credit for it?

Interestingly, early movies (before 1960) listed only the director and actors (and maybe the costume designer if it was a period movie), and these came at the beginning of the film. Someone who worked as an illustrator would never get credit; only the art director might be given onscreen credit. The change to listing everyone who had anything to do with the making of a film was due to trade unions. The unions argued that their members should get credit because it was essential for getting jobs on future films and because non-actor artists should be able to list the movie on their resume. Nowadays, virtually everyone who is in filmmaking belongs to a trade union, which looks out for its members....hence, the 15 minute long credit sequences we see at the end of movies.

In scientific publications, the authors are really the only individuals who receive credit for the work and who can list the paper on their resume, even though a number of non-authors may have contributed to the final product, and in some cases, may have made a critical contribution (for example, a statistician who helped with the experimental design). Such people usually are mentioned only in the acknowledgements, where their contributions are lumped all together in what is essentially a footnote. Because of this, non-authors cannot list the publication on their resume....and can describe only their general role ("In 2012, I assisted in field sampling during a study of coral reefs in Indonesia"). In contrast, a cinematographer, an animator, a sound technician, or even a gaffer can list on their resume the movies they've worked on.

The assignment of credit for a scientific work in part reflects the nature and history of scientific research in which the early discoverer of new knowledge, who was the sole author of the paper, got the credit. This pattern held until the early 1900s when more scientists began collaborating and tackling complex projects requiring teams of people. The numbers of authors on a paper increased while each author's contribution became less clear and quantifiable. Scientific studies increasingly involved technicians and others whose contributions were considered insufficient to count as authors. The tradition of all credit going to part of the scientific team via authorship has persisted despite the increase in non-author contributors to scientific research. Some perhaps suggested the idea for the study; others provided the laboratory space or other essential logistics; many collected data or provided technological expertise; a few gave statistical guidance....none enough to warrant authorship or credit for a scientific discovery but who played a role in the ultimate outcome.

Many scientific publications today are produced by multi-disciplinary teams, each with many people contributing in much the same way as teams of animators or set designers on a film. Papers in some disciplines such as genomics can list 100 authors, a situation that is likely to increase in the future. There is even one recent paper describing the Large Hadron Collider that lists 2,926 authors. In such cases, how do you know who did what and who, other than the first author, made a major contribution? Although some journals (e.g., Nature journals) require an author contributions statement, these are often awkward and potentially lengthy. Would it be more feasible to use a movie credit model in such cases or would that lead to even more confusion?  

Could the movie credit model even work for scientific publications? Well, I can see some problems with it, not the least of which would be resistance from scientists accustomed to getting all the credit. At issue would be who should get credit for the scientific discovery reported in the paper and how would this be indicated in the credits? Of concern, also, might be that someone listed in the credits, but with a minor contribution, would claim a greater role on their CV. I don't think the latter would be any more of an issue than it is in the movie business (a lighting technician is not going to claim she made the movie). In fact, it might reduce this possibility. With the current model of listing multiple authors (using an often unspecified method of assigning position*), minor contributors can get undue credit.

Another stumbling block might be how to sequence the credits to indicate relative contribution. The credits could be listed in a sequence of decreasing contribution, much like they are today in author lists, except that their role would be clear (Principal Investigator, co-Principal Investigator, Laboratory Director, Field/LaboratoryTechnician, Statistician, etc.), and people who often are inappropriately listed as "authors" today would be more accurately credited for their actual role in the study. The Principal Investigator(s) (who conceived and designed the study, conducted the research, and wrote the paper) would be appropriately credited as the discoverer(s) of new knowledge. A movie credit-style listing might add to the length of a paper, but concern about length is likely to be less of an issue as we move to primarily electronic publication of scientific papers and away from printed journals.

The biggest obstacle, perhaps, would be coming up with a standardized system that everyone agrees upon and then how to get everyone to adopt it. It would likely take some of the major journals making the move first and showing how it might work. Several have already implemented the requirement for an author contributions statement, which is an attempt to go in this direction.

I'm sure there would be other objections to the movie credit model, and one could argue that it would not resolve the issue of how to quantify credit for a scientific paper. However, spelling out the roles that each person played might make it easier to quantify credit than using some formula based on the author's position in the author list (as proposed by the paper in PLoS Biology). The other thing such a change would do, which seems to be an improvement over the current model, is to allow those contributors who are now mentioned only in the acknowledgements to claim credit for their contribution to the work.

I'm not necessarily advocating a change from the traditional model to the movie credit model. Possibly, such a change would introduce unforeseen problems and perhaps add to the bickering that already occurs in some research groups as to author sequence and who should get credit for a discovery. However, it's an interesting exercise to contemplate how it might work and who might benefit.

The problem of how to quantify scientific credit is not going to go away and will only worsen as the average number of authors per paper increases. We eventually may be forced to consider a radically different credit system that better reflects the complexity of modern scientific research.

*Someone unfamiliar with how authorship position on scientific papers is determined might assume that the lower the position in the list, the lower the contribution and, thus, the less credit deserved. However, that logic does not always match how author position was determined, which can vary from discipline to discipline and even from lab to lab. Although the first author position is typically occupied by the person who deserves most of the credit (but not always), the ranking of the succeeding authors may have been determined alphabetically, for example, when there are collaborating groups. In some cultures, the last author position frequently goes to the person who deserves substantial credit; this is typically the PI who got the funding and who designed and oversaw the project. In others, the last position may go to the person who contributed the least. Consequently, some may benefit unduly from their position while others may be underrated, depending on how the author sequence is assessed relative to how it was actually determined. 

Image Credit: modified image from morguefile.com