Lie to Me

In the previous post, I posed a hypothetical situation in which a post-doc, Jim, suffers from overconfidence in his abilities, and despite Jim's obvious deficiencies, the lab director thinks he's a valuable contributor to the research project you lead. 

If you've read previous posts, you will recognize that Jim (and possibly the lab director) suffer from the Dunning-Kruger effect. People with this condition are unskilled and incapable of accurately assessing their abilities as well as those of others. The reason is that the skills required to be competent are often the same skills required to evaluate that competence.  They tend to greatly overinflate their performances relative to those of others.  The more ignorant they are, the greater their confidence in themselves.

In the hypothetical situation described above, the project leader is caught between a rock (Jim) and a hard place (lab director).  You have laid out Jim's deficiencies, backed up with ample data, for both Jim and the director...only to have both of them ignore your logical argument and restate their illogical position about Jim's competence.  Their reaction is due to their overall inability to make logical judgments and to recognize that they have this problem.....a vicious cycle that is difficult to break. The more logic you throw at them, the more firmly they adhere to their mistaken opinion and confidence in that decision.

Most of us in science are pretty logical people; so when confronted with such illogical behavior, we go a little nuts.  We simply cannot understand how someone can totally ignore the evidence in front of them.  We make the mistake of thinking that if we just produce enough evidence or a clear-enough argument, that we'll prevail.  However, that is not going to happen with people like this. Think about religious zealots or climate deniers....nothing you say is going to change their minds.  You have to approach such people with a clear understanding of their psychology, not based on your preconceived notions that everyone is as logical as you are.


What does work?  Research has shown that training, for example in social reasoning, helps improve Dunning-Kruger folks' cognitive abilities.  How does that help you, as a project leader? You're not a psychologist; nor do you have time to be training someone who should already have skills in logic.  Quite a conundrum. I don't think there is a really good solution to this situation.

If I had control over this person's position, i.e., was Jim's supervisor, I would place him on probation and clearly outline what outcomes are expected for all tasks, set deadlines for those outcomes, and spell out the consequences of failure. The plan would be detailed in such a way as to leave no wiggle-room for Jim to blame external factors, if he fails.  Both of us must sign this performance plan, acknowledging our understanding of what's expected and the consequences.  If he protests, I would reply that since he is so confident in his abilities, he should have no problem meeting these basic expectations (which I have set up for all others on the team).  If Jim manages to improve and meets his responsibilities, then things will have been improved for the team, and everyone wins.  If Jim fails, then I have documentation to take steps to dismiss him; but I won't feel badly because I've been totally fair and upfront with him.

However, in the hypothetical situation posed, I would not have control over Jim's position, and I also have to deal with the director who suffers from the same cognitive deficit as Jim. In this situation, I would want to get Jim off my team so that his performance no longer harmed me or others.  Here is one possible tactic:  I would suggest to Jim that he should have his own project to run and that he should run the idea by the director.  I would support Jim's proposal when the director asks my opinion.  Convincing the director should not be difficult, given his lack of logic skills and being Jim's biggest fan.

The key will be to have Jim report directly to the director rather than to me.  If the director tries to make me oversee Jim's project, I have two possible responses: 1) I cannot be in charge of Jim's performance unless I'm made his official supervisor (in which case, I can put the previous plan in motion), 2) I state that I have my hands full leading my own project and cannot be expected to handle a second project, and/or 3) I remind him that making me responsible for Jim's project defeats the purpose of giving Jim his own project.  Hopefully, this would not be too much logic for the director to handle.

Assuming I can pull this off, I will no longer have Jim on my team.  Even if this means more work for the team...well, we already were having to redo his work. And the director will eventually see for himself Jim's incompetency. If not, then I might want to start looking for another job--because the director is probably not judging my performance well either.

Some might feel that the above plan is unfair or underhanded.  Actually, this problem with Jim is due entirely to Jim's incompetence and to the director's failure to deal with it.  The two people responsible for the problems the research project is experiencing (mistakes, slow progress, low morale) should have to deal with them, not me and not the other productive team members. The latter situation is what is unfair.  My plan takes the false beliefs of Jim and the director and forces the two of them to deal with the consequences.

So to summarize how to deal with someone suffering from the Dunning-Kruger effect:

1. Don't make the mistake of assuming they think like you do. Trying to use logic doesn't work because they do not think logically.
2. Focus on their overinflated beliefs and take advantage of them in formulating your strategy.
3. Don't cover for them when their incompetency causes problems.
4. Find a way to make them responsible for the consequences of their incompetency.

Image Credit: Jon Wilkins 2011; Creative Commons; http://www.darwineatscake.com

Guilty Until Proven Innocent

My agency has just created an "Office of Science Quality and Integrity".  One of its functions is to oversee how science is carried out and how science products are developed and released to the public.  Many of the protocols revolve around scientific fraud, what constitutes fraud, and the penalties for its commission.  The entire tone of the science integrity rules suggests that fraud is rampant in the agency.  We are guilty until proven innocent.  This approach strikes me as counter-productive--in any profession, but particularly science.  Of any profession, scientists are trained to be unbiased, honest, and meticulous--characteristics that are integral to the performance of science.  It is drilled into us at every stage of training that fraud or bias of any kind is not tolerated.  Yes, there are dishonest scientists, but I think they are rare.  People who are inclined to cheat, cut corners, or commit outright fraud are simply not attracted to the field of science.

So why would so much effort be spent ensuring that government scientists do not commit fraud (or assuring the public of this)?  It may partly be due to the efforts of special interest groups who have attacked scientists and scientific findings (e.g., climate change), charging that the science is poorly done or that scientists are biased. This is a whole other topic, but is what I suspect is behind this renewed emphasis on integrity in science.  Bureaucrats fear embarrassing incidents that lead to Congressional inquiries and funding cuts (the government department that my agency belongs to has been recently embarrassed, although it was a high-level bureaucrat that was responsible--not scientists).  The current administration has also emphasized integrity in science as an important focus, which has been addressed with new rules, greater scrutiny (of scientists), and special offices (to oversee the scrutiny).

Unfortunately, the establishment of an office to oversee scientific integrity, new rules and regulations, and increased scrutiny of science products sends the opposite message.  If scientists are mostly trustworthy and doing their jobs properly, why would there need to be a special office to ensure that our work is free of bias or fraudulent actions? Why the need for new rules and regulations now?  Is the public really assured by the creation of another bureaucracy?

Reading the materials on scientific fraud, one gets the impression that government scientists are ignorant of basic scientific guidelines and need to be given a code of conduct--not what I imagine the rule-makers want to convey. There is an official "Code of Scientific Conduct" for employees in my Department, which has recently been updated and expanded.  It's pretty long, but here are some excerpts (the points that only relate to scientists):

                      (1)        I will place quality and objectivity of scientific and scholarly activities and reporting of results ahead of personal gain or allegiance to individuals or organizations.

                        (2)        I will maintain scientific and scholarly integrity and will not engage in fabrication, falsification, or plagiarism in proposing, performing, reviewing, or reporting scientific and scholarly activities and their products.

                        (3)        I will fully disclose methodologies used, all relevant data, and the procedures for identifying and excluding faulty data.

                        (4)        I will adhere to appropriate professional standards for authoring and responsibly publishing the results of scientific and scholarly activities and will respect the intellectual property rights of others. 

                      (5)        I will welcome constructive criticism of my scientific and scholarly activities and will be responsive to their peer review.

                        (6)        I will provide constructive, objective, and professionally valid peer review of the work of others, free of any personal or professional jealousy, competition, non-scientific disagreement, or conflict of interest.  I will substantiate comments that I make with the same care with which I report my own work.

These are certainly important guidelines, but is there any scientist who's not aware of these basic rules of conduct?  If I were a non-scientist, I would wonder why government scientists must be reminded of these points and, moreover, why the government would employ a scientist who must be reminded that it's wrong to break any of these guidelines. 

Government scientists struggle to keep up with the changing rules.  It's sort of a joke among government scientists that the rules we must follow are moving targets.  Even if you followed the previous rule about something and are caught in between a rule change, you can get your fingers rapped and be required to redo things under the new rule.  The change usually involves a new rule or new step that increases the effort to get something approved....rarely the opposite.  The specific rules guiding the product review process, for example, change constantly, so if your manuscript is caught mid-stream in a change-over, you may be sent back to square one to start all over again.  On one rare occasion, the rule change removed a step: the requirement that abstracts submitted to conferences originally had to go through the same process as manuscripts (2 peer reviews, approvals at multiple levels) was eventually modified to require only supervisor and science unit head approval.  As you might imagine, it was a nightmare trying to get an abstract reviewed and approved in time to meet a conference deadline.

Another concern is the misinterpretation by non-scientists about the ever-evolving body of science.  The mission of science involves exploration, discovery, and risk-taking.  What we report today in journals will likely be modified in the future (or even rejected) as more information becomes available.  Scientists also disagree often about interpretation of results.  Eventually, however, one hypothesis prevails; it stands the test of time while competing hypotheses fall by the way-side, one by one.  However, details continue to emerge from research, which leads to continual modification.

This process is often misunderstood by non-scientists who expect results that are final and written in stone; they interpret any modification of a theory as evidence of wrongdoing by prior researchers.  It's easy to imagine a government study conducted today with current methods and instrumentation is later shown to have been incomplete or even wrong by a future study using a new methodology.  This situation is not only common, but expected in science.  However, scientists don't fault early workers--we usually view them as pioneers, even if their original idea is eventually shown to be incorrect.  Their hypothesis and initial efforts may have opened an entirely new line of research that ultimately led to important discoveries.  Non-scientists (including the media) seem not to understand this.  Special-interest groups have exploited this ignorance and used it to criticize scientists working on controversial topics.  Michael Mann and the Hockey-stick controversy is just one example. 

A change in a scientific concept as more data are collected could be naively interpreted as scientific fraud on the part of the original researcher.  For example, a new study shows conflicting data, which leads to the assumption that the original findings must have been either the result of mistakes or fraud (on the part of the scientist).  Or at least that's what some critics charge...particularly the ones who want to cast doubt on the integrity of scientists and the validity of their work.  That's essentially what happened in Mann's case. 

The creation of doubt in the minds of the public (about a scientific issue) is a powerful strategy that special-interest groups have learned to use. The book, Merchants of Doubt, does a great job of explaining this technique.  It was used by the tobacco industry (it doesn't cause cancer); by critics of the ozone hole and acid rain (they don't exist), by proponents of DDT (it doesn't damage the environment), and by climate deniers (it's not happening).  If you haven't read this book, I highly recommend it.

Audit vs. Accountability

This series of posts is about accountability regimes and their impact on science.  I thought it might be worthwhile at this point to distinguish between audit and accountability.

According to Gaye Tuchman, author of Wannabe U: Inside the Corporate University, not all audits involve accountability and not all audits are coercive.  She gives the example of a university classroom and the imposition of a pop quiz by the professor who just wants to know how well the class is understanding the coursework. If the test scores are not included in the final grade, the test would be an audit. If the professor incorporates the scores in the final grade, she would be imposing accountability in connection with the audit.  Tuchman goes on to give other examples of audits that are clearly cases of accountability: an audit by the Internal Revenue Service could have serious consequences for those who have cheated on their tax return, as well as those who have not kept proper records or whose jobs are eminently auditable (a consultant, for example).  All would be held accountable for any errors, oversights, or deliberate omissions and made to pay a penalty.

The comparable audit vs. accountability situation for scientists might entail a cataloging of the number and quality of publications. If the information is used by the scientist's department simply to promote itself, the publication data is just an audit. If, on the other hand, the data are used to gauge whether the scientist should be promoted (or retained), then the information is being used in an accountability sense.  One amusing example Tuchman describes is for the British system of auditing individual scholarship by professors. The scholars are ranked (by a specific system) on a three point scale and given one, two or three stars reflecting the importance of their work.  This star-ranking system prompted sarcasm by some academic observers, e.g., that scholarship is ranked "in terms of national, international, and inter-galactic importance".  Another system (presumably tongue-in-cheek) proposes a system of scientist ranking using a celebrity-type approach: soap opera stars, other TV stars, Hollywood stars, and Oscar-Emmy winners.

But accountability regimes are not always amusing.  Consider instances in which scientists are called to accountability for findings that certain special interest groups dislike.  More on this later......

The Catch-22 Culture

The accountability fervor that has been increasing in recent years is having an increasingly chilling effect on government science.  In previous posts, I explained how increasing amounts of time are spent by scientists justifying their work and filling out paperwork, attending training courses, and generally jumping through hoops.  In addition to the time taken away from the actual conduct of science, these accountability activities lead to frustration and eventual burnout.  When it becomes so overwhelmingly difficult (and expensive) to get a science product published or to get permission to travel to an international conference, some government scientists will cut way back on their efforts or just give up.     

Let me be clear about what I mean.  I'm not suggesting that there should not be any oversight at all or that government scientists should not be held accountable for their work.  What I'm suggesting is that an "accountability regime", i.e., one in which accountability takes precedence over science mission, will eventually backfire and lead to a workforce that avoids the very activities needed to produce quality science.  One of the reasons is that scientific productivity expectations are pretty low in comparison to academic institutions. For example, in my science unit, PIs are expected to submit only one paper per year (submit, not publish) to be rated fully successful in job performance. Theoretically, one could resubmit the same paper each year and meet the basic requirement.  My point is that when you have low requirements for performance in combination with excessive rules of accountability, there is a great danger that people will decide that the easiest (and safest) route is to do as little as possible. 

Over the past ten years, government scientists have been increasingly audited and subjected to more paperwork affirming that we have not made errors.  To give you an idea of what a government scientist must go through prior to submitting a paper to a journal, here is the path a manuscript now takes (the process has evolved over the past few years).  First, the author submits the manuscript to her supervisor who checks it for technical quality and policy issues and then sends it out for peer review (2 reviews are required).  Depending on how diligent your supervisor is, these reviews may be accomplished in 2 or 3 weeks or languish for months in reviewer purgatory.  Eventually, the reviews come in and they are forwarded to the author for reconciliation. The author must address all comments (no matter how bone-headed they may be) and prepare a reconciliation document detailing how changes were made (or not).  Then the package containing original and revised versions of the ms, the reviews, the reconciliation, and all dated email correspondence are forwarded to the science unit head who goes over everything and approves or disapproves it. If approved, it then goes to the bureau approving official who again goes through the entire package. If your manuscript topic is deemed especially "sensitive", then it undergoes more intense scrutiny.

At any of these higher levels, there may be comments on the technical aspects of the manuscript; in some cases, these comments are helpful, in other cases, not.  In most cases, the officials are not experts in the science topic and may raise inappropriate questions about technical aspects out of ignorance.  Others suggest editing changes that are grammatically incorrect.  Even though the author may be able to answer those questions and explain why a requested change is not correct scientifically or grammatically, the time involved in addressing these various questions adds up.  If the author tries to ignore these, the next official in line will kick back the manuscript and demand all comments be addressed. So the author sometimes spends a lot of time addressing questions and suggested changes that do not improve the manuscript.

Once the bureau official finally signs off, then you are free to submit to a journal where your manuscript will go through the usual gauntlet of editorial and reviewer raking-over-the-coals.  Getting collegial reviews prior to journal submission can be helpful, but all the time and paperwork involved in getting approvals at multiple levels is not.

You may be wondering at this point why should a manuscript that will go through a thorough review by peers and journal editors (i.e., a real review) when it is submitted for publication need to be reviewed beforehand and approved by people who are usually unfamiliar with the field of study?  We ask that question all the time.

The novel "Catch-22" often comes to mind.  This classic novel by Joseph Heller is a critique of bureaucratic logic and operation.  It follows the protagonist, Yossarian, a B-25 bombardier in World War II.  Yossarian is desperate to get out of the war and tries to figure out how to avoid flying missions.  However, the military has a rule, Catch-22, which prevents soldiers from avoiding combat:

"There was only one catch and that was Catch-22, which specified that a concern for one's safety in the face of dangers that were real and immediate was the process of a rational mind.  Orr was crazy and could be grounded. All he had to do was ask; and as soon as he did, he would no longer be crazy and would have to fly more missions. Orr would be crazy to fly more missions and sane if he didn't, but if he were sane he had to fly them. If he flew them he was crazy and didn't have to; but if he didn't want to he was sane and had to."

As Yossarian flies more missions, his commanders keep raising the number of missions required to be discharged from the military. This and other manifestations of the military bureaucracy are variations on the Catch-22 theme.  Enforcers of the crazy rules don't have to prove that their actions against rule violators are actually supported by a provision of the Catch-22.  They can punish violators with impunity.  The ultimate irony (in the novel) is that Yossarian finally realizes that the Catch-22 rule doesn't exist, but because it isn't real, it can't be refuted or overturned.

In the end, what you get is an army (or workforce) that spends all its time following insane rules (or trying to get around them) instead of doing what they were hired to do--fly their missions (or do science).




Catch-22 describes a paradoxical situation in which an individual needs something that can only be had by not being in that situation. For government scientists, the Catch-22 is that we are asked to prove our work is unassailable or essential, but in doing so we demonstrate that there is reason to question it. The example I gave in a previous post was that scientists are required to provide justification for travel to an international conference.  To give a paper is not sufficient. We must demonstrate how our attendance benefits the agency and/or how our failure to attend will have negative consequences (for the agency).

Accountability rules are driven by the fear (of bureaucrats) that a scientific report will turn out to contain a flaw that becomes a public embarrassment or a member of Congress will question travel expenditures.  However, the more we try to "prove" the lack of error, waste, bias, or fraud, the less convincing we are.  It's impossible to prove a negative.  It's the reason our justice system is based on the presumption of innocence (the burden of proof is on the accuser, not the accused).

If there are people who question the validity or integrity of a scientific report, why not require them to prove that there has been error, bias, or fraud? Why put the burden on the scientist to prove they are innocent of the charge, even before the charge has been made? As I suggested above, this burden will eventually chill scientific endeavor, especially for high-profile topics or new research directions.  We've already seen several instances of climate scientists who have been challenged, grilled, and even threatened--and their institutions and agencies have not always leaped forward to support them, and in some cases even fired them.

Instead of promoting openness, accountability regimes create a climate of fear, paranoia, and confusion.  They are antithetical to the mission of science.

Image/video credits: Catch-22 by Joseph Heller (Simon and Schuster);  movie clip from Catch-22 (Paramount Pictures)

The Accountability Regime

This post continues the discussion of the "accountability regime" or "audit society" that many of us find ourselves facing.  See previous post for background.

Just to give you an example of an accountability "rule" that affects me, let me describe what I must do to attend an international conference held on foreign soil.  First, I must get my name on a list well in advance of the date of the conference (months ahead) (and woe be unto you if you submit your name after the deadline for that quarter, for example, in cases where you are invited to speak in a special session only a month in advance by conference organizers).  Then, if more than five people from my agency plan to attend this conference, there must be a special memo requesting permission for our travel and that justifies our specific attendance at such a conference.  This is not just a formality, as we are often warned that "this time" the powers-that-be may deny the request--mainly because they question why scientists should be attending conferences held outside the country in the first place.  In the meantime, we must submit paperwork authorizing our travel in the event we are given permission.  This must be submitted at least 45 days prior to travel. Before I can set foot on foreign soil, I also must receive "country clearance", which is given by the US embassy in that country.  I can only travel with an official passport, which is held under lock and key at the "bureau international office" between trips.

All of these authorizations often do not arrive until the day or so before I'm scheduled to leave (in one case, I was sitting in the airport when the final approval came through--I would have had to cancel otherwise).  This all, as you might imagine, causes extreme stress, mentally and financially.  I must make air and lodging reservations in anticipation of travel--all of which must be canceled in the event my travel is disapproved.  The sums I've already spent on conference registration, lodging reservations, non-reimbursable airfare are lost.  I cannot just go on my personal time--this is not allowed.  Such hassle sort of makes you think twice about attending international conferences, doesn't it?

Why, you might ask, is all of this necessary just to travel to a conference?  The answer is government accountability. It can't appear to the public that government scientists are unnecessarily spending public funds. The paperwork we fill out not only includes an estimate of our travel expenses, but the portion of our salaries covering that time interval.  So a week-long trip might easily add up to $5,000 or more per person, depending on one's salary and the destination.  Ten people attending the same conference would thus incur an amount of $50,000.  Such a sum would sound outrageous to the average citizen, especially one who doesn't get to travel in their jobs.  Never mind that we have such travel budgeted in our grants, and that presenting papers at international conferences is expected of us.  In a few cases, I've had to provide increasingly detailed "justifications" for attendance at international conferences (just giving a paper is not sufficient).

The mind boggles at the money, time, and effort spent by government agencies and other organizations to develop and enforce an "accountability regime".  Why should anyone have to justify activities that are essential to developing or maintaining one's career in education and research--or, from a broader perspective, enhance the scientific credibility of the institution?  But that's what happens when accountability takes precedence over the mission of the organization.