What changed my mind was a course I took (plant biochemistry) in which the professor assigned published articles on a variety of topics to us to read and discuss. Quite a few of the papers involved molecular and other techniques that were as mysterious to me as the process of flying might be to an elephant. Being a conscientious student, however, I endeavored to teach myself how to read and understand scientific papers about unfamiliar topics. To make a long story short, I developed a procedure not unlike the one recently posted by Jennifer Raff, a Research Fellow in Anthropology at the University of Texas (which I'll get to in a moment).
With my method, I was able to effectively critique all the assigned papers and to enthusiastically participate in class discussions (we were graded on our class participation). I discovered that not only could I hold my own with the rest of the class (mostly biochemistry grad students who were more familiar with such articles), I was able to point out critical flaws in the studies that no one else saw. My strong background in statistics was particularly useful in poking holes in experimental designs (to my surprise, most of the assigned papers reported no statistics whatsoever, not even standard error bars). When the professor began calling on me for my assessment of a paper's experimental approach and whether the conclusions were supported by the data, I knew I would never wonder again whether I could understand any scientific paper and whether its findings were valid.
If someone with scientific research training can feel inadequate when faced with an unfamiliar topic, imagine how non-scientists must feel–particularly science writers who are assigned the task of writing a newspaper or magazine article about a new discovery reported in a scientific journal. To help such folks, Jennifer Raff has written a nice blog post called "How to read and understand a scientific article: a guide for non-scientists".
Her post is not only a good guide for non-scientists, it is an excellent primer for science students and even scientific researchers wanting to improve their skills at reviewing papers.
2 comments:
So... what is your method?
As I said in the post, my method was similar to the one described in the referenced post (How to read and understand a scientific paper: a guide for non-scientists).
I essentially wrote out an outline, diagram, or synopsis of the paper (and I avoided reading the abstract so as not to be influenced by it). I wrote down what question(s) were being addressed (even if they were not stated), what experiments were conducted to address each question, what statistics were conducted, what the results showed, and how the authors interpreted those results. This was sometimes a type of flow diagram. I looked to see if the various parts were in agreement or not (for example, the interpretation did not agree with the data). I also wrote down points that were not clear or otherwise identified information that was not given (but was needed to replicate the study). I then compared my synopsis with the abstract. This approach helped me understand what was done, why it was done, whether it was done well, and what it all meant.
I still use this diagraming/outlining approach to review papers, except that I additionally consider certain qualities evident in the paper:
1. Whether it shows a comprehensive and in-depth understanding of the subject matter (based on evidence presented, relevant material cited, knowledge of recent papers) vs. a poor information base and little awareness of recent literature.
2. Whether the paper is convincing in its arguments and shows insight and deep exploration of the data vs. superficial or simplistic arguments and shallow data exploration.
3. Whether the paper stays focused on a central theme or digresses into irrelevant areas.
4. Whether the paper discusses the limitations of the study or instead extrapolates beyond the data and/or speculates about unstudied topics.
5. Whether the paper convincingly and clearly explains how the study advances knowledge or instead uses various "arm-waving" terminology or buzzwords to make the work sound important (or fails to put it into any perspective at all).
So my early method (developed to understand papers in an unfamiliar field) actually formed the basis for how I would later conduct reviews. The key (for me) is writing or drawing things out on paper so that I can see whether and how everything fits together.
Post a Comment