John Bohannon, at io9:
“Slim by Chocolate!” the headlines blared. A team of German researchers had found that people on a low-carb diet lost weight 10 percent faster if they ate a chocolate bar every day. It made the front page of Bild, Europe’s largest daily newspaper, just beneath their update about the Germanwings crash. From there, it ricocheted around the internet and beyond, making news in more than 20 countries and half a dozen languages. It was discussed on television news shows. It appeared in glossy print, most recently in the June issue of Shape magazine (“Why You Must Eat Chocolate Daily”, page 128). Not only does chocolate accelerate weight loss, the study found, but it leads to healthier cholesterol levels and overall increased well-being. The Bild story quotes the study’s lead author, Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health: “The best part is you can buy chocolate everywhere.”
I am Johannes Bohannon, Ph.D. Well, actually my name is John, and I’m a journalist. I do have a Ph.D., but it’s in the molecular biology of bacteria, not humans. The Institute of Diet and Health? That’s nothing more than a website.
Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded…
I got a call in December last year from a German television reporter named Peter Onneken. He and his collaborator Diana Löbl were working on a documentary film about the junk-science diet industry. They wanted me to help demonstrate just how easy it is to turn bad science into the big headlines behind diet fads. And Onneken wanted to do it gonzo style: Reveal the corruption of the diet research-media complex by taking part…
Could we get something published? Probably. But beyond that? I thought it was sure to fizzle. We science journalists like to think of ourselves as more clever than the average hack. After all, we have to understand arcane scientific research well enough to explain it. And for reporters who don’t have science chops, as soon as they tapped outside sources for their stories—really anyone with a science degree, let alone an actual nutrition scientist—they would discover that the study was laughably flimsy. Not to mention that a Google search yielded no trace of Johannes Bohannon or his alleged institute. Reporters on the health science beat were going to smell this a mile away. But I didn’t want to sound pessimistic. “Let’s see how far we can take this,” I said…
What it really proves, of course, is that chocolate is even more popular than fad diets. For that extra fillip of truthiness, the ‘researcher’ (“a general practitioner in on the prank”) actually used bitter chocolate for the ‘trial’. (“When I asked him why, Frank said it was a favorite of the “whole food” fanatics. ‘Bitter chocolate tastes bad, therefore it must be good for you,’ he said. ‘It’s like a religion.'”) All it took was 15 innocent subjects, 3 weeks, and the statisicalistic magic of “p-hacking” — plus a 600-Euro payment to a fake scientific journal (there’s a list of them!) and a press-release blast to get this Totally Real Not Kidding Science-like Study published by a wide range of “professional” magazines and websites always hungry for content…
… People who are desperate for reliable information face a bewildering array of diet guidance—salt is bad, salt is good, protein is good, protein is bad, fat is bad, fat is good—that changes like the weather. But science will figure it out, right? Now that we’re calling obesity an epidemic, funding will flow to the best scientists and all of this noise will die down, leaving us with clear answers to the causes and treatments.
Or maybe not. Even the well-funded, serious research into weight-loss science is confusing and inconclusive, laments Peter Attia, a surgeon who cofounded a nonprofit called the Nutrition Science Initiative. For example, the Women’s Health Study—the largest of its kind—yielded few clear insights about diet and health. “The results were just confusing,” says Attia. “They spent $1 billion and couldn’t even prove that a low-fat diet is better or worse.” Attia’s nonprofit is trying to raise $190 million to answer these fundamental questions. But it’s hard to focus attention on the science of obesity, he says. “There’s just so much noise.”…
Valdivia
Not to be contrarian but this piece is getting a lot of biting commentary from other science writers.
srv
You mean my Green & Black’s chocolate diet is for NOTHING?
I have not been more disappoint since Tim sold me all of that resveratroll stock.
You just can’t trust science.
Ruckus
I know someone who uses chocolate as part of a weight loss diet. My questions as to WTF went unanswered. Now this person is not overweight at all and the rest of the diet is pretty reasonable so no harm no foul but…….
mai naem mobile
I had extended family visiting.From the UK a while ago. One is a diabetic on an insulin pump and he mentioned the portion sizes here made it hard to deal with his pump because they’re so huge here. I bought a silverware set over xmas. I saw the set in the front and figured it was a freebie serving set. Nope, when I got home and opened the box it was an actual place setting. I have been using the teaspoon as a tablespoon. The tablespoons look like big serving spoons. Even bowls and plates are huge. WTF. We don’t need this stuff.
scav
@Valdivia: Poor dears. It’s so unfair to expect them to think critically.
Tissue Thin Pseudonym
@Valdivia: That’s a mighty fine concern troll there. I especially like this bit:
Well, yeah. Of course, the point is that the credibility of science journalists should be damaged, because, as a group, they’re fucking hacks. The author of this response may be one of the responsible ones, but the problem is that there is no way for consumers to tell who the responsible ones are.
Much like the good cops who cover for their dirty brethren, it’s the responsibility of the good science journalists to drive out their bogus counterparts. They’re doing a terrible job of it.
Tree With Water
This is, hands down, the coolest public building I have ever seen in my life. It’s in K.C., of all places. Why haven’t I ever heard of it before?
http://i.huffpost.com/gen/2904412/thumbs/o-KANSAS-CITY-LIBRARY-900.jpg?2
srv
@Tree With Water: Probably because there’s no Ayn Rand in that stack.
Valdivia
@scav: @Tissue Thin Pseudonym:
Seth Mnookin, who teaches Science Writing at MIT and I always thought of as a serious and rigorous science journalist, went on a rant about it, raising exactly the issues in the article I linked. I assumed that he wasn’t just trying to collective cover the asses of the profession. But maybe I got it wrong. I would be curious to ask our own Tom Levenson about it.
Fred Fnord
I feel a little cheated that I never had the opportunity to be taken in by this one.
scav
@Valdivia: I very much miss being able to subscribe to Science News, but I think this opinion rant is ill-advised. I can see making a point that several things were omitted in the IOS article that made it more sensational (not all writers took and ran with the bait) but enough certainly did. If the ‘lesson had been learned” then that shouldn’t have happened to the degree that it did. Why not admit that there are two-bit lazy-ass Science Writers and not whimper about it being shameful to pull the rug out from idiots? Lessons often need to be repeatedly learned.
Stella B
Peter Attia criticizing junk science is amusingly ironic.
Fair Economist
The hoax is brilliant and while the target is science journalists, the misapplication of p values that’s at the heart of the hoax plagues most sciences. This is the reason when Amgen reran studies only about 11% turned out to be reproducible. Intentional fraud is IMO rare but self-delusion is ubiquitous. The problem is that to be a successful scientists you have to have dozens of published papers and it’s basically impossible to find so many genuine results. Scientists *have* to find something to publish so they massage the data until they get something.
Villago Delenda Est
@Fair Economist: Publish or Perish yields unintended consequences. Who would have thunk it?
Tissue Thin Pseudonym
@Fair Economist: Someone really needs to start a serious journal (as in, really peer reviewed and selective) dedicated to publishing only studies with null results. Not only would it allow for more legitimate publication for scientists, it would also serve the purpose of ensuring that null result studies get published prior to the sensational ones that find a spurious effect.
Tissue Thin Pseudonym
Management here turns off the ventilation and most of the lights after 9pm because no one in the building. As the person who is in the building when no one is in the building, that means I’m both insulted and roasting hot.
Note: this is prompted by the clown show post at work at one night a week. The account I work at the other nights, that I really like, provides a small cooler fan for the office over there. One of the many reasons I like it over there.
piratedan
see. this exactly proves Sen. Imhofe’s point….. /////
maybe if these guys did an expose on which field’s scientific claims are dubious and explained why, I might feel like they’ve accomplished something.
Fair Economist
@Tissue Thin Pseudonym: A “Journal of Negative Results” would be a great idea but one journal isn’t nearly enough. The exact numbers depend on how much scientists are relying on multiple comparisons to generate the misleading results, but there are probably multiple unpublished studies for every p<.05 that sees print. You'd need to substantially increase the scientific publication volume.
sheldon vogt
@Tree With Water: KC resident here.
that’s the facade of the parking garage for the downtown library.
hands down, the most impressive public building in the city is the Kauffman Center
for the Performing Arts.
world-class venue
http://www.google.com/imgres?imgurl=http://www.umkc.edu/news/announcements/kauffman_center.jpg&imgrefurl=http://www.umkc.edu/news/announcements/ConservatoryChenYi111.asp&h=258&w=400&tbnid=QDvdmEBY874vRM:&zoom=1&docid=XOFGvyMnZxFfaM&ei=yxZoVdGLBsS-sAWxzoFA&tbm=isch&client=safari&ved=0CC4QMygFMAU
Hkedi
@Tissue Thin Pseudonym: There are a few of those out there already, such as the Journal of Negative Results in BioMedicine . Unfortunately, there are three problems that I see making wide spread adoption difficult. (and this is from somebody with 40-something academic papers)
First, After getting a negative result, it is difficult to justify the time, or get the motivation to write out, proofread, send out, and then modify for questions a paper. Second, negative results almost never get further grant money (the old publish or perish problem). Third, especially for a graduate student, where most research is done, all negative results = no PhD.
Tissue Thin Pseudonym
@Hkedi: That’s a problem with science. If a field doesn’t treat negative results as important, then it’s doing bad science.
TG Chicago
@Tissue Thin Pseudonym: I agree.
And while I’m sure Hkedi is likely correct when saying:
…that’s a problem that goes beyond science. Seems silly to me. X had no effect. Does X-1 have an effect? Let’s find out! That’s good science, I’d think. It’s boring and takes a long time to come upon good findings, but so be it.
But of course, rightwingers would jump all over this and slash and burn. As they regularly do.
Kathleen
OT but I’m worried about my brother and his wife in Dallas. They have four feet of water on the driveway, though no water has crept into the house yet. They have the misfortune of having a creek in back of their house. They can’t leave the house either because of water. 4 inches of rain in one hour. Sheesh.
Valdivia
@Fair Economist:
Similar or perhaps worse problems in something like poli sci no? I don’t know if you have been following the scandal in poli sci about the canvassing for gay marriage paper?
sparrow
It is perfectly possible to publish null results. You might not make it to Nature, and you might not get a lot of publicity, but no journal that I am aware of would prohibit such a publication. The problem is actually bothering to do it — that lies with the scientists. Funny thing, I am publishing a “null result” right now in astronomy. Earlier this year I had been thinking of a project, did some literature search, and found someone else had already tried and gotten a null result. Great! Saved me the trouble of re-doing it.
So at least in astronomy, I don’t think this is as big of an issue. It may be field-specific.
Sm*t Cl*de
@sparrow:
The Journal of Personality & Social Psychology is a major player in social psych. They published Bem’s flawed work on precognition, but refused to even consider publishing a critique / failure-to-replicate explicitly because they were not interested in negative results (with three other top-flight journals rejecting the replication failures for the same reason).
NotMax
It ought to have set off alarm bells from the get-go, as everyone knows that ingestion of chocolate every two hours promotes weight adjustment.
And the more pricey the chocolate, the better.
dr. luba
The New England Journal of Medicine often publishes “negative” results; e.g. new medication/treatment/protocol is NOT superior to older one.
NotMax
@sparrow
Puts me in mind of Peter van de Kamp and the Barnard’s star situtation. (More explanation here.)
Sherparick
Well, you @Valdivia: Yep, a faux study does tend to “shake” up things a bit.
I'mNotSureWhoIWantToBeYet
@Valdivia: That’s a good read. Thanks for the pointer.
Cheers,
Scott.
I'mNotSureWhoIWantToBeYet
@Tissue Thin Pseudonym: Some of the most famous results in physics are null results – e.g. the Michelson-Moreley experiment. There’s nothing that says a journal won’t publish a null-result – a special journal for them isn’t needed.
Doing a null-result experiment well is very, very difficult though…
Cheers,
Scott.
Fair Economist
@sparrow: Yes, there are some fields more willing to publish negative results. Physics does, as does medicine with large randomized trials, as dr luba points out. Some of that is a response to requirements for big expensive studies – those can’t realistically be swept under the rug and replaced. I didn’t know about astronomy; bully for them if they take the more scientific approach. There is also a move underway to improve the situation as a result of the Amgen failure-to-replicate study, but it’s just starting.
@Valdivia: The gay marriage study apparently was a case of *intentional* fraud, which I think is comparatively rare. I agree it reflects the same pressures.
rime
I need to cut calories to lose weight faster
Weight Loss Truth: Cutting your calories down might be a great thing, if you are drastically overeating and stuffing your face. However, if you are eating proportionally then cutting calories might have an aversive affect. If you are cutting calories and are starving your body, then that will lower your metabolism, or in other words slow it down, which may result in you actually not losing any weight at all, even if you are “cutting calories”
For more on what i did to lose weight visit http://tinyurl.com/o46r9lj
Fourten
This made me laugh, audibly. As a fat guy who will, despite some sincere but half hearted efforts, will always be fat any push back on the diet-craze industry is welcome. There is no advise quite so unwelcome as unsolicited diet advise from ppl looking for karma points by trying to correct the most obvious flaws in anyone but themselves.
Now, if you will excuse me, I’m sore from the gym yesterday and another donut will surely make me feel better.
Emily68
@Hkedi: There’s also the Journal of Irreproducible Results. But I wouldn’t take the studies published there too seriously.
dr. luba
The other part of this is the huge number of pay-to-play online journals which are either not peer-reviewed, or only nominally peer-reviewed. Anyone can publish anything somewhere, if they’re willing to cough up the dough, as Bohannon et al did. And the general public, for the most part, does not distinguish between good studies in peer-reviewed journals, and inadequately powered, poorly designed studies in pay to play journals.
It is the job of science journalists to research the research, so to speak, and point out which articles are meaningful, and which aren’t. Unfortunately, many journalists take the same approach to science as they do to politics, which is report “both sides” without performing any analysis,. aka stenography.
grumpy realist
@Hkedi: Well, you can spin it the way I did: I showed that quasi-particles in a frustrated spin-glass was NOT a candidate model to explain high-temperature superconductivity. Got my doctorate on the strength of it, so I guess that’s something…
C.V. Danes
Nice punk job there. Too bad there’s all the real world health issues involved with all those overweight people consuming chocolate thinking it would help them lose weight.
What’s next, a punk study demonstrating how pure cane sugar helps control diabetes? That should wind up being a barrel of laughs!
Uncle Cosmo
One of the dirty little secrets hiding in plain sight in the process of statistical analysis is
–i.e., even if an effect is real (which can only be clear after the study in question is replicated a few times with similar results) its size might well be so small that the actual impact is minuscule & might easily be swamped by other effects.
Unfortunately this is easily disguised by quantitatively incompetent (or flat-out unscrupulous) reporters if the affected population is large enough, e.g., [Insert activity/exposure/substance of choice here] causes 7,000 deaths a year!–which to the typical Homo sap (for whom any number over 999 is essentially “many”) sounds like something to be concerned about–until & unless cooler heads realize that with 7 billion saps on the planet it works out to an additional risk of death of literally one in a million. (By comparison this Guardian article claims worldwide deaths from lightning strikes at 6,000-24,000 per year, though the link to the numbers is dead.)
This truism ought to be as well known as “correlation =/= causality” but isn’t.
I vaguely recall a book (whose name I’ve forgotten) about the (mis)use of statistics whose author stated that “statistically significant” results from biostatistical analysis wouldn’t get his attention unless the reported odds ratio was a least 2. IMO that’s a bit of over-reaction, but I sympathize with his attitude.
lawguy
@Valdivia: A as in ONE science writer?
les
@Tree With Water:
It’s a parking ramp, actually. But across the street from the library, which is pretty cool in its own right.
Bill Murray
@Uncle Cosmo:
I think you mean any number over 20 or 21 for men.
@dr. luba:
I would say all journals are pay to play, the difference is who pays. Copyright transfer and in some journals treating the paper as a work-for-hire are often abusive.
Uncle Cosmo
@Bill Murray:Dunno about that. Plenty of dimbulbs out there know their hunnerts from green slips of paper featuring Ben Franklin. (Though they’re not real sure which President he was.) But how many know who appears on the US$1000 note? (I had to gargoyle it myself…)