Kevin Drum mulls an annoying trend in social science research and harrumphs.
Call me naive, but WTF? I have no training at all, and I’m keenly aware of the problems Gelman is talking about. How is it possible to complete a PhD program and not have this kind of thing drilled into your consciousness for all time? Can there really be people out there who are being trained that “statistically significant” = real, and nothing more? It’s mind boggling. Are there any PhD programs out there that would would fess up to this?
I attended a scientific Bachelor’s program and earned graduate degrees in a few different fields, so I might as well weigh in here. Short answer, yes. There are a lot of them. Long answer, it depends on what field of science you have in mind. In ‘descriptive’ fields like ecology where manipulating the thing you study is hard to impossible, students tend to walk away with a grasp of statistics that is comprehensive and exacting. Most will learn the proper use of complex software packages like SAS that let you apply many levels of analysis to huge data sets. On the other hand laboratory fields often hand out Ph.D.s to people who only know to run a two-tailed t-test in Excel, maybe, and give a blank uncomprehending stare if you suggest they test for outliers or apply post-hoc standards to a hypothesis that they thought up after the data came in. You tend to see that in laboratory disciplines, where more control over conditions lets you narrow the scope of experiments to a couple of variables at most.
In the end it always comes down to what reviewers expect. Professional scientists have to keep a lot of balls in the air at once. Publish or perish is a real thing that has a potent ability to focus the attention. Most people thus do not have the energy to learn statistical rules that have no bearing on whether your paper or grant will get accepted. During my oceanography years I got raked over the coals for some pretty arcane rules of correlation and inference, so I cracked the books in order to move my career forward. In my life as a cell biologist I have published in second-tier Nature journals and PNAS, several times each, and nobody has yet demanded anything more demanding than printing my significance values in a larger font size. How did I get those numbers? Should I have used an ANOVA or a post-hoc test? Whatever it’s all good. People in each field learn as much about statistics as they need to keep publishing and scoring grants. For us lab rats the important stuff could fit on the margins of Arthur Laffer’s cocktail napkin.
Then again cell biology is a relatively ‘hard’ field with a lot of turnover in areas with enough interest to make the big journals. BS tends to get caught and refuted pretty fast. Thus you can half-ass something and see it published, but most people don’t.
Regarding sociology, I read most of that stuff for entertainment value only. In most cases there are just too many variables for anyone to control. If something comes up that has an unusually strong and consistent result such as the Stanford marshmallow experiment or the Milgram prison fiasco, or if a number of independent groups consistently report the same thing, then I tend to pay attention. Other stuff occupy that dangerous nexus between easy to do, no best practices yet and very hard to control properly. fMRI is my prime candidate for emergency bathroom reading only, followed by any field with -omics in it that is younger than five years old. Best practices take a little while to develop. Any field of ‘science’ practiced by MD’s is a tetchy topic for me at least; everyone knows about the professional rivalry between MD’s and the other kind of doctor, and I am a Ph.D. Let’s leave it at that.
Leo Artunian
I assume the misspelling “pele” for “people” was occasioned by the preceding “scientists have to keep a lot of balls in the air at once.” Goal!
BGinCHI
Still, your average sociology major is like Einstein compared to a business major.
Amir Khalid
Enticed by the headline, I read through this post and there was no porn at all. I am disappoint.
@Leo Artunian:
Pele good, Maradona better, George Best!
Cassidy
There is another kind of doctor other than an MD/DO?
BGinCHI
@Amir Khalid: You see Liverpool play Preston? Some good young talent….
Violet
Ain’t that the truth.
Amir Khalid
@Cassidy:
I once met a software company CEO who turned out to be a diploma-mill doctor.
Amir Khalid
@BGinCHI:
Missed that one, but I’m hearing good things about the new lads.
Tim F.
@Cassidy: Doctor Who. Doctor Evil did not attend three years of evil medical school to be called Mister. Pretty sure that Doctor Strange does not have a medical license. Same for Doctor Doom and Doctor Octopus.
Herbal Infusion Bagger
Stanford prison experiment, although AFAIK Zimbardo’s results weren’t replicated in other studies: maybe Stanford students are just bigger a-holes than the general population. Milgram’s was “will people torture others when told to” experiment.
Herbal Infusion Bagger
If an MD has seen one cases of a certain set of symptoms/or treatment working: “In my practice, I have observed the following…”
If an MD has seen two cases: “I have repeatedly seen in my practice the following…”
If an MD has seen three cases: “Over and over again I have seen that…”
BruinKid
@Tim F.: What about Dr. Hook and his Medicine Show?
Seanly
Yeah, Drum falls for the oldest canard in the book – all scientists are equally trained on all topics. Just like all Southern accents sound the same.
Also, while the social sciences are important, I would never assume off the top of my head that there are as rigorous in statistical methodology as the more ‘hard’ sciences. Economics, socialogy, pyshcology, etc. aren’t about hard & fast rules that always apply.
Poopyman
Dr. Joyce Brothers haz a sad.
schrodinger's cat
Physicists don’t need no damn statistics apart from calculating the margin of error for their results.
Tim F.
@Herbal Infusion Bagger: Heh heh.
Just looked it up: Doctor Strange does have a medical degree. Victor Von Doom embellished his resume and Dr. Octavius is/was some sort of physicist. Asking what exactly Dr. Who has a doctorate in would apparently destroy the universe.
schrodinger's cat
Also too, economists don’t understand how percentages work, so expecting them to be sophisticated statisticians is a bit much.
RSA
I’ve mentioned this before, but once when I was in an online argument with an MD about creationism, I thought to google Are doctors scientists? The top hit was an editorial in the British Medical Journal titled Doctors Are Not Scientists, which I thought was a nice find. It begins, “Some doctors are scientists—just as some politicians are scientists—but most are not.”
BruinKid
Yeah, I was consulting for a Ph.D. student in Education over at our rival school, and the sociological analysis he needed involved a chi-squared test. But his advisor specifically told him to run ANOVA, even though the numbers in the table cells were integer scores so that the normality condition did not hold, and so ANOVA was worthless. I told him this, but his advisor basically didn’t care, and just wanted the ANOVA results.
All right, fine. To be nice, I ran both tests, and showed him both sets of results. Turns out there were cases where ANOVA rejected the null, but the chi-squared test didn’t, and vice versa. And yet they’re publishing the (obviously wrong) ANOVA results.
Robert Sneddon
@Tim F.: Call a senior consultant “Doctor” Over Here in the UK and expect to get your balls/ovaries handed to you on a shish kebab. It’s “Mister” aka “Master” (or if you’re going GRRM, “Maester”). Pond scum medical school graduates with only a few years residency under their belt are doctors.
Suzanne
Social sciences are totally different animals. I realize that everyone wants hard data from double-blind studies all the time, but that isn’t possible all the time. And just because we can’t accurately measure something doesn’t mean it isn’t real or “for entertainment value only”. There is always more in heaven and earth than is dreamt of in your (Doctor of) Philosophy.
Quicksand
Dr. Pepper is delicious! (p<0.05)
ArchTeryx
I admit, I’m one of the guilty ones when it comes to figuring out statistics in my science, being a molecular biologist and virologist. I’ve a severe allergy to higher math and it’s the reason I ended up in an empirical science in the first place — I’d started out as a biochemist, but the studies there are MUCH more in-depth math wise.
But one benefit is that it straddles the fence between extremely quantitative sciences and the “soft” sciences like social sciences, so one develops a sympathy for both. One of my most important papers came from an observation that statistically was insignificant, but to my eye, something was going on. We developed a better system to test the hypothesis, and all of of a sudden, we got REALLY significant data. Sometimes, it’s all in how you ask the question.
Tim F.
@Robert Sneddon: does that mean Dr. Who must automatically be a doctor of medicine?
khead
I will go with Doctor Detroit. Because he had hookers.
dr. luba
Keep getting a big, red, scary malware warning from Safari when I try to visit this page. What is up with that? Has Newsmax gone rogue?
ArchTeryx
@Tim F.: I always got the feeling he called himself “The Doctor” to mess with the muggles, and as a point of pride: he fixed things, whatever those things happened to be.
BruinKid
@dr. luba: Wow, same here, just now. Hasn’t happened before. WTF? (And I’m using Firefox.)
Gravenstone
@Tim F.:
Actually, Doctor Strange does had an MD. He was a successful neurosurgeon who turned to the arts arcane after an accident.
BruinKid
@Quicksand: Heh.
Gravenstone
@Tim F.: Since River Song tells us that the human race took the name Doctor to mean healer explicitly from The Doctor, then yes. he’s clearly a medical doctor. *nods*
Chris
@Tim F.:
And then there’s Doctor Jones, who’s tied with his father, Doctor Jones, in awesomeness points.
Gravenstone
@BruinKid: Same here with Firefox. Couldn’t override or back out so had to reload the site from the bookmark. I suspect the “warning” is itself an example of malware.
schrodinger's cat
OT :
This oven that we are in makes me miss winter
* Thats my backyard after the massive snow storm last February, and yes I was complaining about the snow back then.
schrodinger's cat
Speaking of fictional doctors, I liked the holographic Doctor from Voyager.
Amir Khalid
@Gravenstone:
I just got the malware warning too. Maybe we need mistermix to look into this.
scav
Also depends the crowd the social / lab / experimental etc crowd the various scientists are running with. The Behavioral Geographers in the depts I attended were right in there with the Oceanographers / Climatologists / Biogeographers / Remote Sensors / Economic Geographers in math/model/stat geektitude and just having to sit through dept-wide colloquia necessarily exposed one to a stunning variety of tests and measures. Repeated Measures Manovas to Spatial Heteroskedasticity in error terms to EDA all paraded past. And we also saw some “soft” approaches to hard topics (cultural / historical / linguistic definitions of ‘forest’ and how that interacted with on the ground landscape preservation in India). There’s not as simple boundary between hard and soft sciences as it might appear, there are also harder and softer approaches (more or less appropriate to the exact problem at hand) to “harder” and ”softer” issues.
gene108
@Suzanne:
Why can’t a social scientist blind the people he’s studying twice?
cthulhu
Oh I totally agree, and I say that as someone who has published several papers using fMRI…
Chris
@schrodinger’s cat:
Or – DAMN IT, JIM! – his all-too-real counterpart in the first show.
scav
@Gravenstone: But there are also those peoples for whom doctor means warrier so I think its also definitional-weffinitional, and that’s before we even explore the bearded parallel-Us.
Leo Artunian
Herbal Infusion Blogger: “Stanford prison experiment, although AFAIK Zimbardo’s results weren’t replicated in other studies: maybe Stanford students are just bigger a-holes than the general population.”
I was at Stanford about that time and yes, some of them certainly were. Yes indeedy.
scav
@gene108:
Budget cuts. Re-sighting the subjects that you can double-blind them is increasingly pricey.
Citizen_X
What about Doctor Love?
JCT
Ah yes, the never-ending “MD-taint” – those of us with both degrees know it well. And I’m not going to play tit-for-tat here, but I and a fair number of my colleagues also publish in good journals, have 10+ years of sustained R01 funding and also take care of patients, most of them high-risk and thus requiring specialized care. And what do you know, for those of us with backgrounds in genetics, we actually have to have a reasonable grasp of statistics (yes, beyond a Chi-square).
But one thing is true , MDs doing research are going the way of the dodo so soon this over-generaliztion will die out. And then I will have to continue to review grants from my only-PhD colleagues that are rife with bizarre assumptions about semi-magical approaches for new disease therapeutics. And their stats can be lousy too.
Not picking on you, really Tim F. but what is gained by perpetuating these stereotypes?
@Suzanne: Actually, the lack of double-blind (prospective) studies is a significant problem in biomedical research as well, often due to the problems in generating a large enough enrollment and ethical concerns. Actually a huge problem in my field where the so-called treatment “guidelines” are largely based on anecdote and association. A big problem.
Tim F.
@JCT:
Pride, mostly. Like many postdocs I am (was until recently) in my mid 30’s and could theoretically make more money working for tips.
Herbal Infusion Bagger
Heh. Although almost all of the Stanford alums I’ve met have been great. I’d be happy working for a Stanford alum.
But I never, ever, ever would willingly work for a Harvard B-school alum again. I think if Zimbardo had done the experiment with Harvard B-school students, they’d have ground up the prisoners and marketed them as pet food within 48 hours.
RSA
I forgot to mention this:
My area of expertise used to be exploratory data analysis, and I’m a duffer when it comes to experiment design and analysis, but I think that reporting effect sizes goes a long way toward fixing this problem. I might build a fancy system that affects the flipping of a ferrous coin, for example (not possible, though, as far as I know), and show that when it’s turned on the coin comes up heads significantly more often than in a non-biased environment. But if the effect size is small enough (.500000000001 versus .499999999999), no one will care.
JCT
@Tim F.:
Yes, the post-doc situation is a travesty. FWIW, though not remotely as bad, my full-time colleagues in my clinical specialty easily make 3-4X as much as I do as a full professor at a state university. But that way madness lies.
Herbal Infusion Bagger
@JCT:
An attempt to counteract the hype that results when some numbskull at the NYT/Time/Cosmo.HuffPo pulls out a single unreplicated study out of some low-impact medical journal that Substance/Livestyle/Environmental Condition X causes or cures cancer/heart disease/diabetes/erectile dysfunction/etc. Crappy science journalism isn’t the fault of the research MD community, but publication of poorly designed studies doesn’t help.
JCT
@Herbal Infusion Bagger: I’m not so sure that only MD-driven research is the problem here. Seems to me there have been some pretty high-profile papers that hit the popular press (cloning comes to mind) that have turned out to be dead wrong. For the Journals involved, including Science/Nature/Cell – that can be the price for going after the “sexy” stuff.
No scientist is immune from wanting to make a splash, and while I would never defend the popular press (I spend most of every family get-together debunking what people read in reputable newspapers) – this is a complex problem. We fight about it all the time on the clinical research side where it is nearly impossible to design “great” studies – too many variables. So we train our fellows to read critically and try their best not to over-extrapolate. The differences turn out to be very nuanced and while I can train a junior clinician to understand that, the popular press has to, excuse the term, dumb things down for a general audience and the nuance disappears.
This is not unlike the near-nightmare unleashed when drug companies could advertise new drugs on the sides of buses. Explaining to a patient that the data really didn’t show a difference between the new drug and their old standby is a hard sell…..
Those “poorly-designed” studies went through peer review as well – getting $$ for clinical studies is no mean feat. I’ve been on a translational study section where each grant is assigned a basic scientist, a clinically-oriented scientist and a statistician. And I’m sure some lousy studies got through even that gauntlet .
No easy answers.
JR in WV
Tim,
I fearlessly explored and indeed got the dread malware warning which took me to this site:
https://www.stopbadware.org/firefox?hl=en-US&url=http%253A%252F%252Fwww.balloon-juice.com%252F2013%252F07%252F16%252F136456%252F
Since I had other options, I didn’t try to do anything while there. I had BalloonJuice up from earlier today, and when I pressed refresh, I got the site, strangely altered. There was only the main list of posts/threads. No ads, no contact list, no blogs list, nothing!
There was nothing down either side, just the main list of posts down the middle (which was OK by me). But when I clicked on the 50 shades post, that’s when things went south.
I’m not afraid of possible malware, I’m running on Linux… but it is a concern that the web site is booked into a malware list by Google, apparently.
BruceJ
As a rule, in a lab setting, if you really have to start applying more exotic statistical tests than a standard 2-tailed T-Test you either a) don’t have enough data or b) need a better experimental design to magnify the range of results you get. More dots and a wider spread on the graph paper. It’s not rocket science, even statistically. :-)
All those “exotic” statistical tests folks who do field biology know, they know because they’ve got a big enough ‘n’ to apply the tests in a realistic and proper manner.
What I’d prefer that every PhD student have drilled into their heads was enough readings from Tufte so their damned charts make frikken sense. In particular, I’d require any graduate to be able to recite or paraphrase large sections of “The Visual Display of Quantitative Information” from memory.
Harumph!
chopper
@Chris:
doctor octagon FTW.
kuvasz
Tim, you are making a big mistake with the statement, “I am a Ph.D.”
You are not. You have a Ph.D.
I don’t want to go all Wizard of Oz on you, but the diploma is a piece of paper, just like the Scarecrow was handed. However, it represents the efforts and learned experience of the process.
Just such situation once happened to me on a job interview. I had handed my current business card to the initial interviewer, an Indian national, and Ph.D., who noted that the card did not mention my own Ph.D., (or MS). I told him that as my current job as a technical service director I frequently visited textile and polymer extrusion plants to trouble-shoot my products and had found that having been identified as a “Ph.D.” the mill/plant workers, with barely a 12th grade education, would feel uncomfortable with speaking to one to solve the current problem I came to fix.
The interviewer got upset that I was not promoting myself as a “Ph.D. To which I responded, as above, to his strong displeasure. The fellow based his entire existence on being “a Ph.D.” and I simply brushed off being one.
I turned down the job offer, too.