In a story about a new watchdog catching a serial fabricator in Italy, the journal Nature included some startling allegations (bolding mine).
in 2008, [Enrico] Bucci, a molecular biologist, founded BioDigitalValley in Pont Saint Martin, Italy. Its services include pulling out all published images of gel-electrophoresis analysis — which separates and identifies large molecules such as proteins and sequences of RNA — that are relevant to a particular disease or tissue.
Bucci and his team created a database hosting all accessible biomedical papers published since 2000.
[…] The list ran to more than one million, so he looked only at Italian scientists. Using in-house software that could isolate images of, for example, gels, and check them for simple features such as possibly duplicated portions, he ran an automatic check of all the papers the Italian researchers had published. He focused on highly cited researchers for whom the automatic check had revealed multiple papers with anomalous images. […] Now midway through the analysis, he estimates that around one-quarter of the thousands of papers featuring gels that he has analysed so far potentially breached widely accepted guidelines on reproducing gel images. And around 10% seem to include very obvious breaches, such as cutting and pasting of gel bands. Some journals were more affected than others, he says. Those with a high impact factor tended to be slightly less affected.
A month ago The Economist posed a troubling riddle: why do more and more scientific studies fail to reproduce? To answer that puzzle you might ask Enrico Bucci or other other self-appointed watchdogs like the now-defunct Science Fraud blog or the psychology-oriented “data whisperer”. It is easy enough to crap on the usual range of science critics that we link around here. However, unlike creationists and climate whores these guys do a real service by policing science from the inside.
As for why we need these guys in the first place, I think it is essential to understand a little bit about real scientists and the human beings who do it. Science is about learning things, but the unit of progress is the peer-reviewed paper. We measure each other by our papers, we use them to determine whether a student is ready to graduate and, with grant success, we use paper output to choose whether junior faculty have done enough to earn tenure. Your ideas can thus be brilliant or a bit dull but they only matter if you can package them in a format that a meaningful journal is willing to publish. It thus gets more and more frustrating when you approach what feels like the finish line for a paper and one important control or line of inquiry refuses to work.
Now, here is the line that separates believers from the kind of skeptical mindset that is too rare even among scientists. Most of us have a story that we plan to tell before we pick up a pipettor. In fact hypothesis testing being the soul of good inquiry, it takes a pretty poor scientist to go into a study without an outcome that matches some combination of what they know and what they hope to be true. Some few of us even have the combination of foresight and luck to finish a paper without leaving some or all of that original story in bin where trash goes, along with a bit of your ego. And that is good! If we knew what would happen before we did it then we would be engineers.
The ego thing is not a trivial problem. A lot of people, too many, will over invest in an idea that they published already, dig their trenches too deep in a dispute with competing labs, or they just hate to let go of that project diagram they sketched out on day one. Of course these problems get worse the closer you get to the finish line. After the other data all seemed to fall in place without any shoehorning, that one last misbehaving control experiment can drive you batty. Sometimes the equipment really is misbehaving, but sometimes that niggling problem points to an alternative story that you either overlooked or refused to consider*. Everyone wants to be rigorous, but just as much you need to get that paper out NOW NOW NOW because [X] is ONE MONTH AWAY, where [X] is a grant deadline, deadline for revising a paper, graduation date, conference presentation, faculty review, postdoctoral job interview or any other of the countless urgent events that litter scientific life.
We often feel too rushed to step back and take a hard skeptical look at that misbehaving control experiment, but we need to do it anyway. The big risk of course being that when you ‘make it fit’ you might guess wrong. Then, at best, someone fails to replicate the work and the offender moves on. In a sense that is part of the back and forth that has characterized science since forever. On the other hand folks like the offender in the article above cannot or will not walk away, and these guys end up with the Marmion problem (“Oh what a tangled web we weave…”): they have to play increasing games with their data to keep up their side of the argument. I sincerely hope that folks like Bucci will short-circuit those destructive cycles and, better yet, convince would-be fudgers to step back and deal with their annoying experiments before they make that first mistake.
(*) Not that you will hear me complain. The latter left an opening in my field that I drove some very high-profile papers through. Never underestimate the value of looking at a problem through semi-naive eyes.
pamelabrown53
I think the real problem is who and how research $$$ are granted and distributed. Since our tax structure was decimated by the 2 Bush 2 tax cuts, it seems that any real science has to be done on a shoestring budget and covertly.
greennotGreen
I read the Economist article, and I thought it was a bit too negative. Science is self-correcting (although it may take longer than it should, sometimes.) Somebody screws up a series of experiments either stupidly or fraudulently; if they get the results published, no one will be able to replicate it, and it comes to a dead end. That’s why how many times a paper is cited has become increasingly important.
I will say that we’re doing a piss-poor job of basic science education, though. I work at a highly competitive bio-medical research center/ graduate and medical school. I have only a B.A. and a B.S. (but two decades plus of lab work,) yet I am amazed at how many graduate students and post docs come in who don’t know how to design a simple experiment. They must believe “control” is only a key on a keyboard.
different-church-lady
Or, in pictorial form…
Capri
To be fair, gels are probably an area that allows for this more than any other. The little stripes can be quite subjective, particularly if someone wants to see them.
I wouldn’t take the issues with gel images and paint all scientific research that way.
It’s widely known that any published “representative sample” is likely the very best image or result the researcher obtained, but that rarely affects the results.
I once read that peer-reviewed scientific research is very similar to the English gentleman culture. Pretty much all interaction between groups is formalized, and a scientist’s reputation and word is all important.
Big R
Tim, what was the hole in your field?
RareSanity
As an engineer, not sure if I agree with this categorization.
Once we get finished dealing with the constant requirement creep of the client, the absurd delivery schedule set forth by sales, and the need to also make whatever it is “sizzle” by marketing…there’s many a sleepless night thinking, “How the hell are we going to pull this off?”
Just because we know how something is supposed to look/operate/feel before starting, doesn’t mean we know how we’re going to get there.
That’s why there’s an ‘R’ in R&D…there is a research component to our jobs as well. Granted we are not usually researching new knowledge. But we are researching ways to apply existing knowledge in ways, or under constraints, never done before…mainly “faster and cheaper”, with more blinkenlights.
People love them some blinkenlights…
dollared
I think this is the same as the Slate effect. As market pressures increase, so does the pressure to produce extraordinary (in Slate terms “counterintuitive”) results. It’s not enough to do good science, you have to do groundbreaking, breathtaking, previously unheard of results.
And just as most extra-ordinary results in journalism fail the common sense test, most extra-special results in science are, well, not reproducible. But they might get you really good grants during the five years it takes to yield that disappointing conclusion….
sm*t cl*de
Ironic, somehow, that the editors of ‘Nature’ are the worst foot-draggers when it comes to retracting papers when they’re shown to be fraudulent.
http://retractionwatch.com/category/by-journal/nature-retractions/
aimai
My father, who has a Nobel, told me years ago that there was a tiny but persistent cottage industry in science (and he has published in physics, biology, and chemistry) of people who fake replicating other people’s work and publish approving studies demonstrating that the first study, the more important study, has been confirmed. They draft off other people’s research, content with second rank but secure lives. This was in answer to the question of why it sometimes took a long time for even important work to be disconfirmed.If you are very quiet and persistent about it, at least in the old days, you could get up a nice little record of publishing and never have to even do the experiments at all.
Mullah DougJ
It’s got to be more than one-in-four, no? That seems awfully low to me.
Joel
No love for Retraction Watch?
Halteclere
@RareSanity:
Also an Engineer here. I think a better differentiation between Engineers and Scientists (besides that Scientists discover knowledge, Engineers apply that knowledge), is that Engineering is about using ideas to find a way to achieve a specific result, while Science uses results to prove ideas.
Crusty Dem
@Mullah DougJ:
Agreed. This is 1-in-4 for a subset of figures that are easily altered but also easily checked (and this fraud detection will only catch a subset of fraud – people who cheat by misloading gels, applying incorrect Abs or other non-digital manipulation will not be caught).
In my field, you could never determine if a manipulation was real or not. Only exact replication of an experiment could determine if it was correct and even that wouldn’t prove fraud. Additionally, my previous bosses wouldn’t even allow me to report irreproducibility of previously published, high profile work for fear of offending future grant reviewers.
In short: the system is fucked, top to bottom.
sw
If you are a real researcher, you are doing your research in a logical manner,following a well thought out research plan that is designed to explore some specific issue. Papers should be generated organically as part of the research process. This is how we communicate our research results. Serious introspection by working researchers at all levels will result in the conclusion that a huge amount of resources are wasted doing essentially useless crap because it is perceived to be likely to lead to ‘a pretty good paper’. It doesn’t take much self awareness to understand that this is getting the process backwards. But when job security depends on getting those ‘decent papers’ to the exclusion of the over-all research effort, human nature will prevail.
Roger Moore
I think the number one reason is statistics. Plenty of researchers don’t really understand statistics, so they are more than happy to report things that pass an arbitrary threshold even when that threshold is misapplied, e.g. not adjusting a significance threshold when testing multiple hypotheses. If we want less junk science, we should require any paper that involves statistics to consult with an actual statistician.
Origuy
Tim, does peer review include the software used to process the data? In software engineering, there is (or should be) a lot of focus on code walkthroughs and unit testing. I worked in research years ago as an undergrad, and I doubt that the researchers made the code I wrote available to reviewers. I remember discussions a few years ago in Scienceblogs about a bug in someone’s code that invalidated his conclusions. From what I’ve been able to see, a lot of research groups rely on one-off code that may not have been fully tested or inspected.
Roger Moore
@sw:
I don’t think that’s completely true. There’s often a branch point in research where you switch gears from discovering stuff to proving a point to readers, especially editors and reviewers. The process of wrangling a batch of experiments into shape for presentation to others often requires additional work that isn’t strictly necessary for ongoing research but which makes a stronger paper. For example, you might want to re-run some early experiments with the latest version of your methods so you can use a consistent methodology across all the stuff discussed in your paper. That isn’t strictly necessary for research, but it simplifies the paper and avoids wrangling with reviewers and confusing readers. I find papers where people have bothered to do that kind of polishing more helpful than ones that are more about following the research as it happened, including all the messy details.
Roger Moore
@Origuy:
It damn well ought to. Whenever I’ve written a paper that involved coding, I’ve made the code available to reviewers and offered it to readers. When I’ve reviewed articles that involve code, I’ve been allowed to get a copy (generally by downloading from somewhere online, so my anonymity as a reviewer is preserved) so I can see what I think of it. And the papers always have to describe what’s supposed to be going on under the hood, even if the implementation is sometimes buggy.
Howard Beale IV
I saw a posting from a retired shrink who was posting a segment of a
very serious articlepaid pharma paper where he slyly inserted a Dread Scott reference in the middle of a paragraph.Wonder how many papers that have been auto-processed have those kind of ‘features’….
Roger Moore
@Howard Beale IV:
Paging Alan Sokal…
Crusty Dem
@Origuy:
@Roger Moore:
Absolutely not. Seriously. Additionally, most code is written by the scientists themselves with a completely variable level of coding experience and is never checked by anyone else. And most labs treat every line of code as a trade secret and don’t share it with anyone under any circumstances – to the level that most people don’t even bother to ask about whether they could use your code. I wrote some basic software for data analysis a half decade ago that filled a niche that no commercial software had, no one even asked me for it…
dmbeaster
Its not just bogus data. There are all sorts of reasons why even the most credible scientists cut corners in their research papers, and falsify information. I am reminded of the controversy concerning Robert Gallo, his 1984 paper to Nature, and the discovery of AIDS. Any credible reading of that controversy results in the conclusion that although Gallo was and is a great scientist who did seminal work concerning AIDS, he was also more than willing to be dishonest about the discovery when “discovery” equaled very big $$$$ and patent rights.
Howard Beale IV
@Joel: Bookmarked.
Crusty Dem
@Joel:
I’m pretty sure i first heard about Retraction Watch here. Quality.
Joel
This is a subject that I think about a lot, as a researcher who often grapples with reproducing other people’s work. Some day, I’d like to put together some long, thought-out piece on it, but the issue that I have with present-day science has less to do with fraud — which, by its intentional nature, has to be known by at least one person — and more to do with ignorance, carelessness, or just plain laziness.
Mistakes happen, I get that, but a fair amount of research is just wrong. Not because of any malicious intent, but because the authors just had NFI what they were doing. Ultimately this does get ferreted out and corrected, but it takes a long time and a lot of work. And errors get compounded even further through the citation tree. Once something wrong becomes part of a (minor) dogma, then good luck trying to get to the root of the problem. The proliferation of journals and the emphasis on paper volume isn’t helping matters, either.
Original Lee
Carl Djerassi had it right in Cantor’s Dilemma, and it’s kind of scary that the dynamics haven’t changed much.
Part of the problem is that proofing somebody else’s work is not sexy, therefore not as worthy of funding as startling new ideas. Experimental labs, especially at universities, used to be able to spend some money on using graduate students to replicate experiments. This was also kind of a weeding process – does this person have “good hands”? I’m not sure how much of that is done now.