• Menu
  • Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Before Header

  • About Us
  • Lexicon
  • Contact Us
  • Our Store
  • ↑
  • ↓
  • ←
  • →

Balloon Juice

Come for the politics, stay for the snark.

Teach a man to fish, and he’ll sit in a boat all day drinking beer.

Israel is using food as a weapon of war. Unforgivable.

Motto for the House: Flip 5 and lose none.

Technically true, but collectively nonsense

Putin dreamed of ending NATO, and now it’s Finnish-ed.

Republican also-rans: four mules fighting over a turnip.

A sufficient plurality of insane, greedy people can tank any democratic system ever devised, apparently.

Too often we hand the biggest microphones to the cynics and the critics who delight in declaring failure.

My years-long effort to drive family and friends away has really paid off this year.

Compromise? There is no middle ground between a firefighter and an arsonist.

There are a lot more evil idiots than evil geniuses.

Historically it was a little unusual for the president to be an incoherent babbling moron.

Bogus polls are all they’ve got left. Let’s bury these fuckers at the polls a year from now.

These days, even the boring Republicans are nuts.

“Can i answer the question? No you can not!”

Just because you believe it, that doesn’t make it true.

Sitting here in limbo waiting for the dice to roll

“woke” is the new caravan.

Jack be nimble, jack be quick, hurry up and indict this prick.

White supremacy is terrorism.

A snarling mass of vitriolic jackals

Putin must be throwing ketchup at the walls.

And now I have baud making fun of me. this day can’t get worse.

When do the post office & the dmv weigh in on the wuhan virus?

Mobile Menu

  • Four Directions Montana
  • Donate with Venmo, Zelle & PayPal
  • Site Feedback
  • War in Ukraine
  • Submit Photos to On the Road
  • Politics
  • On The Road
  • Open Threads
  • Topics
  • COVID-19 Coronavirus
  • Authors
  • About Us
  • Contact Us
  • Lexicon
  • Our Store
  • Politics
  • Open Threads
  • 2024 Elections
  • Garden Chats
  • On The Road
  • Targeted Fundraising!
You are here: Home / Science & Technology / Do one in four scientific papers contain bogus data?

Do one in four scientific papers contain bogus data?

by Tim F|  December 4, 20134:29 pm| 26 Comments

This post is in: Science & Technology

FacebookTweetEmail

In a story about a new watchdog catching a serial fabricator in Italy, the journal Nature included some startling allegations (bolding mine).

in 2008, [Enrico] Bucci, a molecular biologist, founded BioDigitalValley in Pont Saint Martin, Italy. Its services include pulling out all published images of gel-electrophoresis analysis — which separates and identifies large molecules such as proteins and sequences of RNA — that are relevant to a particular disease or tissue.

Bucci and his team created a database hosting all accessible biomedical papers published since 2000.

[…] The list ran to more than one million, so he looked only at Italian scientists. Using in-house software that could isolate images of, for example, gels, and check them for simple features such as possibly duplicated portions, he ran an automatic check of all the papers the Italian researchers had published. He focused on highly cited researchers for whom the automatic check had revealed multiple papers with anomalous images.

[…] Now midway through the analysis, he estimates that around one-quarter of the thousands of papers featuring gels that he has analysed so far potentially breached widely accepted guidelines on reproducing gel images. And around 10% seem to include very obvious breaches, such as cutting and pasting of gel bands. Some journals were more affected than others, he says. Those with a high impact factor tended to be slightly less affected.

A month ago The Economist posed a troubling riddle: why do more and more scientific studies fail to reproduce? To answer that puzzle you might ask Enrico Bucci or other other self-appointed watchdogs like the now-defunct Science Fraud blog or the psychology-oriented “data whisperer”. It is easy enough to crap on the usual range of science critics that we link around here. However, unlike creationists and climate whores these guys do a real service by policing science from the inside.

As for why we need these guys in the first place, I think it is essential to understand a little bit about real scientists and the human beings who do it. Science is about learning things, but the unit of progress is the peer-reviewed paper. We measure each other by our papers, we use them to determine whether a student is ready to graduate and, with grant success, we use paper output to choose whether junior faculty have done enough to earn tenure. Your ideas can thus be brilliant or a bit dull but they only matter if you can package them in a format that a meaningful journal is willing to publish. It thus gets more and more frustrating when you approach what feels like the finish line for a paper and one important control or line of inquiry refuses to work.

Now, here is the line that separates believers from the kind of skeptical mindset that is too rare even among scientists. Most of us have a story that we plan to tell before we pick up a pipettor. In fact hypothesis testing being the soul of good inquiry, it takes a pretty poor scientist to go into a study without an outcome that matches some combination of what they know and what they hope to be true. Some few of us even have the combination of foresight and luck to finish a paper without leaving some or all of that original story in bin where trash goes, along with a bit of your ego. And that is good! If we knew what would happen before we did it then we would be engineers.

The ego thing is not a trivial problem. A lot of people, too many, will over invest in an idea that they published already, dig their trenches too deep in a dispute with competing labs, or they just hate to let go of that project diagram they sketched out on day one. Of course these problems get worse the closer you get to the finish line. After the other data all seemed to fall in place without any shoehorning, that one last misbehaving control experiment can drive you batty. Sometimes the equipment really is misbehaving, but sometimes that niggling problem points to an alternative story that you either overlooked or refused to consider*. Everyone wants to be rigorous, but just as much you need to get that paper out NOW NOW NOW because [X] is ONE MONTH AWAY, where [X] is a grant deadline, deadline for revising a paper, graduation date, conference presentation, faculty review, postdoctoral job interview or any other of the countless urgent events that litter scientific life.

We often feel too rushed to step back and take a hard skeptical look at that misbehaving control experiment, but we need to do it anyway. The big risk of course being that when you ‘make it fit’ you might guess wrong. Then, at best, someone fails to replicate the work and the offender moves on. In a sense that is part of the back and forth that has characterized science since forever. On the other hand folks like the offender in the article above cannot or will not walk away, and these guys end up with the Marmion problem (“Oh what a tangled web we weave…”): they have to play increasing games with their data to keep up their side of the argument. I sincerely hope that folks like Bucci will short-circuit those destructive cycles and, better yet, convince would-be fudgers to step back and deal with their annoying experiments before they make that first mistake.

(*) Not that you will hear me complain. The latter left an opening in my field that I drove some very high-profile papers through. Never underestimate the value of looking at a problem through semi-naive eyes.

FacebookTweetEmail
Previous Post: « Just…Wonderful — xkcd edition/afternoon open thread
Next Post: On Orchestrating Less Violence »

Reader Interactions

26Comments

  1. 1.

    pamelabrown53

    December 4, 2013 at 4:46 pm

    I think the real problem is who and how research $$$ are granted and distributed. Since our tax structure was decimated by the 2 Bush 2 tax cuts, it seems that any real science has to be done on a shoestring budget and covertly.

  2. 2.

    greennotGreen

    December 4, 2013 at 4:50 pm

    I read the Economist article, and I thought it was a bit too negative. Science is self-correcting (although it may take longer than it should, sometimes.) Somebody screws up a series of experiments either stupidly or fraudulently; if they get the results published, no one will be able to replicate it, and it comes to a dead end. That’s why how many times a paper is cited has become increasingly important.

    I will say that we’re doing a piss-poor job of basic science education, though. I work at a highly competitive bio-medical research center/ graduate and medical school. I have only a B.A. and a B.S. (but two decades plus of lab work,) yet I am amazed at how many graduate students and post docs come in who don’t know how to design a simple experiment. They must believe “control” is only a key on a keyboard.

  3. 3.

    different-church-lady

    December 4, 2013 at 4:57 pm

    Or, in pictorial form…

  4. 4.

    Capri

    December 4, 2013 at 5:06 pm

    To be fair, gels are probably an area that allows for this more than any other. The little stripes can be quite subjective, particularly if someone wants to see them.
    I wouldn’t take the issues with gel images and paint all scientific research that way.

    It’s widely known that any published “representative sample” is likely the very best image or result the researcher obtained, but that rarely affects the results.

    I once read that peer-reviewed scientific research is very similar to the English gentleman culture. Pretty much all interaction between groups is formalized, and a scientist’s reputation and word is all important.

  5. 5.

    Big R

    December 4, 2013 at 5:10 pm

    Tim, what was the hole in your field?

  6. 6.

    RareSanity

    December 4, 2013 at 5:12 pm

    If we knew what would happen before we did it then we would be engineers.

    As an engineer, not sure if I agree with this categorization.

    Once we get finished dealing with the constant requirement creep of the client, the absurd delivery schedule set forth by sales, and the need to also make whatever it is “sizzle” by marketing…there’s many a sleepless night thinking, “How the hell are we going to pull this off?”

    Just because we know how something is supposed to look/operate/feel before starting, doesn’t mean we know how we’re going to get there.

    That’s why there’s an ‘R’ in R&D…there is a research component to our jobs as well. Granted we are not usually researching new knowledge. But we are researching ways to apply existing knowledge in ways, or under constraints, never done before…mainly “faster and cheaper”, with more blinkenlights.

    People love them some blinkenlights…

  7. 7.

    dollared

    December 4, 2013 at 5:16 pm

    I think this is the same as the Slate effect. As market pressures increase, so does the pressure to produce extraordinary (in Slate terms “counterintuitive”) results. It’s not enough to do good science, you have to do groundbreaking, breathtaking, previously unheard of results.

    And just as most extra-ordinary results in journalism fail the common sense test, most extra-special results in science are, well, not reproducible. But they might get you really good grants during the five years it takes to yield that disappointing conclusion….

  8. 8.

    sm*t cl*de

    December 4, 2013 at 5:19 pm

    Ironic, somehow, that the editors of ‘Nature’ are the worst foot-draggers when it comes to retracting papers when they’re shown to be fraudulent.

    http://retractionwatch.com/category/by-journal/nature-retractions/

  9. 9.

    aimai

    December 4, 2013 at 5:20 pm

    My father, who has a Nobel, told me years ago that there was a tiny but persistent cottage industry in science (and he has published in physics, biology, and chemistry) of people who fake replicating other people’s work and publish approving studies demonstrating that the first study, the more important study, has been confirmed. They draft off other people’s research, content with second rank but secure lives. This was in answer to the question of why it sometimes took a long time for even important work to be disconfirmed.If you are very quiet and persistent about it, at least in the old days, you could get up a nice little record of publishing and never have to even do the experiments at all.

  10. 10.

    Mullah DougJ

    December 4, 2013 at 5:22 pm

    It’s got to be more than one-in-four, no? That seems awfully low to me.

  11. 11.

    Joel

    December 4, 2013 at 5:25 pm

    No love for Retraction Watch?

  12. 12.

    Halteclere

    December 4, 2013 at 5:30 pm

    @RareSanity:

    And that is good! If we knew what would happen before we did it then we would be engineers.

    As an engineer, not sure if I agree with this categorization… That’s why there’s an ‘R’ in R&D…there is a research component to our jobs as well. Granted we are not usually researching new knowledge. But we are researching ways to apply existing knowledge in ways, or under constraints, never done before…mainly “faster and cheaper”, with more blinkenlights.”

    Also an Engineer here. I think a better differentiation between Engineers and Scientists (besides that Scientists discover knowledge, Engineers apply that knowledge), is that Engineering is about using ideas to find a way to achieve a specific result, while Science uses results to prove ideas.

  13. 13.

    Crusty Dem

    December 4, 2013 at 7:08 pm

    @Mullah DougJ:

    Agreed. This is 1-in-4 for a subset of figures that are easily altered but also easily checked (and this fraud detection will only catch a subset of fraud – people who cheat by misloading gels, applying incorrect Abs or other non-digital manipulation will not be caught).

    In my field, you could never determine if a manipulation was real or not. Only exact replication of an experiment could determine if it was correct and even that wouldn’t prove fraud. Additionally, my previous bosses wouldn’t even allow me to report irreproducibility of previously published, high profile work for fear of offending future grant reviewers.

    In short: the system is fucked, top to bottom.

  14. 14.

    sw

    December 4, 2013 at 7:28 pm

    If you are a real researcher, you are doing your research in a logical manner,following a well thought out research plan that is designed to explore some specific issue. Papers should be generated organically as part of the research process. This is how we communicate our research results. Serious introspection by working researchers at all levels will result in the conclusion that a huge amount of resources are wasted doing essentially useless crap because it is perceived to be likely to lead to ‘a pretty good paper’. It doesn’t take much self awareness to understand that this is getting the process backwards. But when job security depends on getting those ‘decent papers’ to the exclusion of the over-all research effort, human nature will prevail.

  15. 15.

    Roger Moore

    December 4, 2013 at 7:37 pm

    why do more and more scientific studies fail to reproduce?

    I think the number one reason is statistics. Plenty of researchers don’t really understand statistics, so they are more than happy to report things that pass an arbitrary threshold even when that threshold is misapplied, e.g. not adjusting a significance threshold when testing multiple hypotheses. If we want less junk science, we should require any paper that involves statistics to consult with an actual statistician.

  16. 16.

    Origuy

    December 4, 2013 at 8:03 pm

    Tim, does peer review include the software used to process the data? In software engineering, there is (or should be) a lot of focus on code walkthroughs and unit testing. I worked in research years ago as an undergrad, and I doubt that the researchers made the code I wrote available to reviewers. I remember discussions a few years ago in Scienceblogs about a bug in someone’s code that invalidated his conclusions. From what I’ve been able to see, a lot of research groups rely on one-off code that may not have been fully tested or inspected.

  17. 17.

    Roger Moore

    December 4, 2013 at 8:18 pm

    @sw:

    If you are a real researcher, you are doing your research in a logical manner,following a well thought out research plan that is designed to explore some specific issue. Papers should be generated organically as part of the research process.

    I don’t think that’s completely true. There’s often a branch point in research where you switch gears from discovering stuff to proving a point to readers, especially editors and reviewers. The process of wrangling a batch of experiments into shape for presentation to others often requires additional work that isn’t strictly necessary for ongoing research but which makes a stronger paper. For example, you might want to re-run some early experiments with the latest version of your methods so you can use a consistent methodology across all the stuff discussed in your paper. That isn’t strictly necessary for research, but it simplifies the paper and avoids wrangling with reviewers and confusing readers. I find papers where people have bothered to do that kind of polishing more helpful than ones that are more about following the research as it happened, including all the messy details.

  18. 18.

    Roger Moore

    December 4, 2013 at 8:25 pm

    @Origuy:

    Tim, does peer review include the software used to process the data?

    It damn well ought to. Whenever I’ve written a paper that involved coding, I’ve made the code available to reviewers and offered it to readers. When I’ve reviewed articles that involve code, I’ve been allowed to get a copy (generally by downloading from somewhere online, so my anonymity as a reviewer is preserved) so I can see what I think of it. And the papers always have to describe what’s supposed to be going on under the hood, even if the implementation is sometimes buggy.

  19. 19.

    Howard Beale IV

    December 4, 2013 at 8:27 pm

    I saw a posting from a retired shrink who was posting a segment of a very serious article paid pharma paper where he slyly inserted a Dread Scott reference in the middle of a paragraph.

    Wonder how many papers that have been auto-processed have those kind of ‘features’….

  20. 20.

    Roger Moore

    December 4, 2013 at 8:34 pm

    @Howard Beale IV:

    Wonder how many papers that have been auto-processed have those kind of ‘features’….

    Paging Alan Sokal…

  21. 21.

    Crusty Dem

    December 4, 2013 at 8:44 pm

    @Origuy:
    @Roger Moore:

    Absolutely not. Seriously. Additionally, most code is written by the scientists themselves with a completely variable level of coding experience and is never checked by anyone else. And most labs treat every line of code as a trade secret and don’t share it with anyone under any circumstances – to the level that most people don’t even bother to ask about whether they could use your code. I wrote some basic software for data analysis a half decade ago that filled a niche that no commercial software had, no one even asked me for it…

  22. 22.

    dmbeaster

    December 4, 2013 at 9:00 pm

    Its not just bogus data. There are all sorts of reasons why even the most credible scientists cut corners in their research papers, and falsify information. I am reminded of the controversy concerning Robert Gallo, his 1984 paper to Nature, and the discovery of AIDS. Any credible reading of that controversy results in the conclusion that although Gallo was and is a great scientist who did seminal work concerning AIDS, he was also more than willing to be dishonest about the discovery when “discovery” equaled very big $$$$ and patent rights.

  23. 23.

    Howard Beale IV

    December 4, 2013 at 9:07 pm

    @Joel: Bookmarked.

  24. 24.

    Crusty Dem

    December 5, 2013 at 12:19 am

    @Joel:

    I’m pretty sure i first heard about Retraction Watch here. Quality.

  25. 25.

    Joel

    December 5, 2013 at 1:53 pm

    This is a subject that I think about a lot, as a researcher who often grapples with reproducing other people’s work. Some day, I’d like to put together some long, thought-out piece on it, but the issue that I have with present-day science has less to do with fraud — which, by its intentional nature, has to be known by at least one person — and more to do with ignorance, carelessness, or just plain laziness.

    Mistakes happen, I get that, but a fair amount of research is just wrong. Not because of any malicious intent, but because the authors just had NFI what they were doing. Ultimately this does get ferreted out and corrected, but it takes a long time and a lot of work. And errors get compounded even further through the citation tree. Once something wrong becomes part of a (minor) dogma, then good luck trying to get to the root of the problem. The proliferation of journals and the emphasis on paper volume isn’t helping matters, either.

  26. 26.

    Original Lee

    December 5, 2013 at 2:14 pm

    Carl Djerassi had it right in Cantor’s Dilemma, and it’s kind of scary that the dynamics haven’t changed much.

    Part of the problem is that proofing somebody else’s work is not sexy, therefore not as worthy of funding as startling new ideas. Experimental labs, especially at universities, used to be able to spend some money on using graduate students to replicate experiments. This was also kind of a weeding process – does this person have “good hands”? I’m not sure how much of that is done now.

Comments are closed.

Primary Sidebar

Recent Comments

  • Martin on Righteous Rant Open Thread (Apr 15, 2024 @ 6:48pm)
  • Kay on Righteous Rant Open Thread (Apr 15, 2024 @ 6:46pm)
  • Martin on Righteous Rant Open Thread (Apr 15, 2024 @ 6:37pm)
  • Martin on Righteous Rant Open Thread (Apr 15, 2024 @ 6:34pm)
  • WaterGirl on Righteous Rant Open Thread (Apr 15, 2024 @ 6:29pm)

🎈Keep Balloon Juice Ad Free

Become a Balloon Juice Patreon
Donate with Venmo, Zelle or PayPal

Balloon Juice Posts

View by Topic
View by Author
View by Month & Year
View by Past Author

Balloon Juice Meetups!

All Meetups
Talk of Meetups – Meetup Planning
Proposed BJ meetups list from frosty

Fundraising 2023-24

Wis*Dems Supreme Court + SD-8
Virginia House Races
Four Directions – Montana
Worker Power AZ
Four Directions – Arizona
Four Directions – Nevada

Featuring

Medium Cool
Artists in Our Midst
Authors in Our Midst
Positive Climate News
War in Ukraine
Cole’s “Stories from the Road”
Classified Documents Primer

Calling All Jackals

Site Feedback
Nominate a Rotating Tag
Submit Photos to On the Road
Balloon Juice Mailing List Signup
Balloon Juice Anniversary (All Links)
Balloon Juice Anniversary (All Posts)

Fix Nyms with Apostrophes

Balloon Juice for Ukraine

Donate

Twitter / Spoutible

Balloon Juice (Spoutible)
WaterGirl (Spoutible)
TaMara (Spoutible)
John Cole
DougJ (aka NYT Pitchbot)
Betty Cracker
Tom Levenson
David Anderson
Major Major Major Major
ActualCitizensUnited

Political Action 2024

Postcard Writing Information

Balloon Juice for Four Directions AZ

Donate

Balloon Juice for Four Directions NV

Donate

Site Footer

Come for the politics, stay for the snark.

  • Facebook
  • RSS
  • Twitter
  • YouTube
  • Comment Policy
  • Our Authors
  • Blogroll
  • Our Artists
  • Privacy Policy

Copyright © 2024 Dev Balloon Juice · All Rights Reserved · Powered by BizBudding Inc

Share this ArticleLike this article? Email it to a friend!

Email sent!