In a story about a new watchdog catching a serial fabricator in Italy, the journal Nature included some startling allegations (bolding mine).
in 2008, [Enrico] Bucci, a molecular biologist, founded BioDigitalValley in Pont Saint Martin, Italy. Its services include pulling out all published images of gel-electrophoresis analysis — which separates and identifies large molecules such as proteins and sequences of RNA — that are relevant to a particular disease or tissue.
Bucci and his team created a database hosting all accessible biomedical papers published since 2000.[…] The list ran to more than one million, so he looked only at Italian scientists. Using in-house software that could isolate images of, for example, gels, and check them for simple features such as possibly duplicated portions, he ran an automatic check of all the papers the Italian researchers had published. He focused on highly cited researchers for whom the automatic check had revealed multiple papers with anomalous images. […] Now midway through the analysis, he estimates that around one-quarter of the thousands of papers featuring gels that he has analysed so far potentially breached widely accepted guidelines on reproducing gel images. And around 10% seem to include very obvious breaches, such as cutting and pasting of gel bands. Some journals were more affected than others, he says. Those with a high impact factor tended to be slightly less affected.
A month ago The Economist posed a troubling riddle: why do more and more scientific studies fail to reproduce? To answer that puzzle you might ask Enrico Bucci or other other self-appointed watchdogs like the now-defunct Science Fraud blog or the psychology-oriented “data whisperer”. It is easy enough to crap on the usual range of science critics that we link around here. However, unlike creationists and climate whores these guys do a real service by policing science from the inside.
As for why we need these guys in the first place, I think it is essential to understand a little bit about real scientists and the human beings who do it. Science is about learning things, but the unit of progress is the peer-reviewed paper. We measure each other by our papers, we use them to determine whether a student is ready to graduate and, with grant success, we use paper output to choose whether junior faculty have done enough to earn tenure. Your ideas can thus be brilliant or a bit dull but they only matter if you can package them in a format that a meaningful journal is willing to publish. It thus gets more and more frustrating when you approach what feels like the finish line for a paper and one important control or line of inquiry refuses to work.
Now, here is the line that separates believers from the kind of skeptical mindset that is too rare even among scientists. Most of us have a story that we plan to tell before we pick up a pipettor. In fact hypothesis testing being the soul of good inquiry, it takes a pretty poor scientist to go into a study without an outcome that matches some combination of what they know and what they hope to be true. Some few of us even have the combination of foresight and luck to finish a paper without leaving some or all of that original story in bin where trash goes, along with a bit of your ego. And that is good! If we knew what would happen before we did it then we would be engineers.
The ego thing is not a trivial problem. A lot of people, too many, will over invest in an idea that they published already, dig their trenches too deep in a dispute with competing labs, or they just hate to let go of that project diagram they sketched out on day one. Of course these problems get worse the closer you get to the finish line. After the other data all seemed to fall in place without any shoehorning, that one last misbehaving control experiment can drive you batty. Sometimes the equipment really is misbehaving, but sometimes that niggling problem points to an alternative story that you either overlooked or refused to consider*. Everyone wants to be rigorous, but just as much you need to get that paper out NOW NOW NOW because [X] is ONE MONTH AWAY, where [X] is a grant deadline, deadline for revising a paper, graduation date, conference presentation, faculty review, postdoctoral job interview or any other of the countless urgent events that litter scientific life.
We often feel too rushed to step back and take a hard skeptical look at that misbehaving control experiment, but we need to do it anyway. The big risk of course being that when you ‘make it fit’ you might guess wrong. Then, at best, someone fails to replicate the work and the offender moves on. In a sense that is part of the back and forth that has characterized science since forever. On the other hand folks like the offender in the article above cannot or will not walk away, and these guys end up with the Marmion problem (“Oh what a tangled web we weave…”): they have to play increasing games with their data to keep up their side of the argument. I sincerely hope that folks like Bucci will short-circuit those destructive cycles and, better yet, convince would-be fudgers to step back and deal with their annoying experiments before they make that first mistake.
(*) Not that you will hear me complain. The latter left an opening in my field that I drove some very high-profile papers through. Never underestimate the value of looking at a problem through semi-naive eyes.