In yesterday’s thread Dog Dawg Damn made an intriguing statement on causal inference and implicitly about ethics:
Retrospective, not Randomly controlled, and not strong evidence. I’ve seen better studies show no effect on same question
I would love to see them write an Institutional Review Board application for a randomized control trial (RCT) — actually I would love to see the revised IRB application after the IRB sends the initial application back for substantial revision along with a note to review the Helsinki principles —-
16. In medical practice and in medical research, most interventions involve risks and burdens.
Medical research involving human subjects may only be conducted if the importance of the objective outweighs the risks and burdens to the research subjects.
19. Some groups and individuals are particularly vulnerable and may have an increased likelihood of being wronged or of incurring additional harm.All vulnerable groups and individuals should receive specifically considered protection.
33. The benefits, risks, burdens and effectiveness of a new intervention must be tested against those of the best proven intervention(s), except in the following circumstances:
Where no proven intervention exists, the use of placebo, or no intervention, is acceptable; or
Where for compelling and scientifically sound methodological reasons the use of any intervention less effective than the best proven one, the use of placebo, or no intervention is necessary to determine the efficacy or safety of an intervention
and the patients who receive any intervention less effective than the best proven one, placebo, or no intervention will not be subject to additional risks of serious or irreversible harm [DMA Emphasis added] as a result of not receiving the best proven intervention.
Extreme care must be taken to avoid abuse of this option.
Maybe they can write an IRB application that addresses these substantial considerations. I have a damn hard time imagining how to do that but then again my imagination is limited.
More notably, not everything can be randomized due to ethical, legal or pragmatic reasons. We have massive fields which use naturally occurring experiments that create substantial variation after an intervention occurs in some places, people or times and when there is nothing new happening. Using appropriate methods while acknowledging the assumptions needed to make causal inferences allows us to make sense of the world while establishing cause and effect.
In the past five years, I’ve used inverse probability weighing matching, instrumental variable approaches, fixed and random effect models, difference in difference, synthetic controls, synthetic difference in difference, stacked difference in difference, regression discontinuity, difference in discontinuity, and matched border pair designs to make causal estimates of policies. The methodologists have even more variants and options that I am not even hinting at. Some of those papers have been published, others are under review at the moment and more are being written. All of these methods have substantial assumptions but conditional on those assumptions being well supported, we can say that X caused Y.
We can say that things result from interventions even without a randomized experiment of kicking people off Medicaid for shits and giggles or exposing them to $0 or $1 health insurance plans or randomly assigning states or people within states to Healthcare.gov or a state based marketplace. The first is unethical as hell as we know health insurance has substantial mortality effects. The second requires an act of Congress or at least a state legislature and the last is HOW THE HELL DO WE ACTUALLY IMPLEMENT THAT ONCE WE GET AN ACT OF CONGRESS WITHOUT CONTROL/TREATMENT GROUP CONTAMINATION.
When feasible and ethical, randomization is absolutely wonderful. We should use randomized trials as much as we can as it addresses unbalanced confounding. But for a wide variety of reasons, randomization is sometimes not feasible or ethical so we should figure out how to see the world as it is with appropriate methods. And a staggered stacked difference in difference is a very good method for this context.
Baud
Baud! 20XX!: More Randomized Human Medical Trials!
Another Scott
This is excellent. (I could suggest putting the Helsinki stuff in a blockquote box or otherwise setting it off, but that’s a minor quibble.)
Most researchers try really hard to get this stuff right, because (among other things) nobody likes their work having a zillion cites for being wrong. That’s why it can take months to get a paper together to even have it ready to send out for review. That’s why coauthors can argue over some conclusion being too strong or that a different flavoring word is needed instead. Etc. Doing this stuff well is hard work.
And that’s before one gets to the legal and ethical issues of working on humans!
Thanks again.
Cheers,
Scott.
sab
Thank you so much for this post.
Maxim
Thank you.
Wag
My medical intellectual heart is entirely in favor of Evidence Based Medicine, with it’s attendant placebo controlled randomized clinical trials as the strongest of all possible medical evidence, HOWEVER, my brain knows enough to realize that some things simply cannot be studied via an RCT, and the EBM adherents insistence on the RCT as being the only worthwhile evidence is wrongheaded and narrow minded. With this example you demonstrate how their insistence would harm trans youth. In the COVID world we see EBM zealots insisting that we cannot mandate masks or vaccines because no one performed the required double blind placebo controlled randomized clinical trial. This despite deep evidence of the effectiveness of both interventions, and the impossibility of ever being able to perform said trials.
Their insistence on perfection misses the point and actively harms society.
stinger
Thank you, David.
Fake Irishman
As a political scientist, I co-sign this blog post.
Bill Arnold
I am reminded of (haunted by) the trajectory of anti-masking in the COVID-19 pandemic, and not just in the USA.
Early in the COVID-19 pandemic, the more disciplined anti-maskers were quite confident that there would never be population-level RCTs of the effects of mask wearing on SARS-CoV-2 spread.
There were even anti-mask propagandists saying this in weak moments; that we will never be able to ethically know whether population-level mask wearing can lower spread in a deadly respiratory virus pandemic, so therefore, we should not do so.
Then that large Bangladesh study emerged[1]. It was a repurposed study of how social interventions could increase mask wearing in a population. (I expect that the submarine in the linked paper was intended, but haven’t pursued.)
The anti-maskers then resorted to mischaracterization, belittling the moderate decrease in spread of COVID-like disease while neglecting to mention that “that mask wearing averaged 13.3% in villages where no interventions took place but increased to 42.3% in villages where in-person interventions were introduced.”, that the increase in masking was only 30 percent and a reduction in COVID-19 spread was still observed.
Also, that “COVID-like” disease was being used as a proxy for COVID-19. (Insufficient PCR testing, in rural Bangladesh, with COVID-19 spreading rapidly and killing widely.)
So IMO, people who make these arguments (that only RCTs count) are often not arguing in good faith.
Amusingly, most academic literature on mask-wearing vs infection by respiratory viruses is now 2020 and later. Still, we continue to see the insufficient RCTs nonsense. That recent Cochrane review is a prime example of such nonsense. (Also agenda-driven, pretty sure.)
[1] Impact of community masking on COVID-19: A cluster-randomized trial in Bangladesh (Abalick et al, 2 Dec 2021)
Anonymous At Work
I hate hate hate Kennedy Krieger decision about the JHU IRB lead abatement trial.
I would cite Kennedy Krieger as a reason that no IRB would ever approve such an application.
David Anderson
@Anonymous At Work: Reading through some of the commentaries about Kennedy Krieger — there is a lot of questions about the value of counterfactuals, second best solutions and political institutions.
owlbrick
Zing!
Anonymous At Work
@David Anderson: It’s the treatment of a consent form as a legal contract. Move from there to “How do we get the money to do the necessary lead abatements?” answered with “Show how bad it is.” I know that’s somewhat the path of Tuskegee (read Bad Blood by Jim Jones) took with syphilis, but the judge’s opinion went down lines that create massive problems for science and IRBs. You can’t invest wisely either in science or using science unless it is good science. And the judge starting with the study results and working backwards is bad science.
Victor Matheson
Yeah, I read that comment, and I immediately thought of one of my favorite (no, really!) Donald Rumsfeld quotes (or a bastardization of it, actually).
“You go to war with the data you have, not the data you wish you had.”
Randomized control trials might be the gold standard, but you may not have that data, you may find it unethical to collect that data, you might not have the time or money to run that trial, or a RCT might simply be impossible to run given the question you are trying to ask. (Try running an RCT for professional sports stadium subsidization (my area of research) for example.)
The American Economic Review is the most prestigious journal on the planet (although to be fair the papers and proceedings issue is not subject to the same level of review as all of the other papers.) The paper looked pretty good to me
(And yeah, a million thumbs up for this follow-up post.)
C Stars
I’ll echo everyone else in saying thank you for this. There seems to be no end to calls for more research and evidence, when in fact the treatment protocol in place is the one that has been found to be the most effective and beneficial to patients. I also think that folks like that commenter, who feel for some inscrutable reason that they have the ethical authority to weigh in on other people’s medical decisions, likely don’t even understand what treatment looks like for trans kids. People seem to think that tweens can walk into a doctor’s office and walk out the same day popping hormone pills. Not at all our family’s experience, even with a very liberal healthcare provider in a very liberal place. Our care team is thoughtful and deliberate and over seven years of seeing my kid have recalibrated their recommendations based on the latest research and their own knowledge of and frequent interaction with their patient. In all of their interactions (which includes mandatory check-ins with their own in-house pediatric gender therapist) their main concern is ensuring that their patients are supported at home and at school, and that dysphoria never becomes enough of an issue that self-harm ideation can take hold. Sadly, losing trans patients to suicide is common enough that many of these practitioners have experienced it.
But even if trans tweens were walking out of clinics popping hormone pills, it really wouldn’t be that different from current treatment of non-trans tweens. A doctor reflexively prescribed me birth control pills when I was 13 because I had painful periods. I suffer no ill effects from this, decades later. Similarly, I know that cis boys have been prescribed testosterone for “delayed puberty” since at least the 70s. And trans people have been taking cross hormones since at least the 40s! This is not a new treatment.
/rant!
RSA
Nice post, David. You’ve reminded me of a famous historical controversy featuring Ronald Fisher, sometimes called the father of statistics: Does smoking cause cancer? (Doubtless you know the story; this article tells it well.)
https://priceonomics.com/why-the-father-of-modern-statistics-didnt-believe/
David Anderson
@Bill Arnold: That Abaluck paper was massively impressive and good work.
Llelldorin
Wait—randomized trials aren’t good enough. For real research, you have to randomly throw some people off Medicaid for shits and giggles but not tell either them or their doctors that you’ve done so!
(Seriously, thank you for the post.)
cmorenc
While I too am rooting for injuries and / or indictments of Trump and DeSantis, I cannot help recallng the proverb: “be careful what you wish for, you may get it” – meaning that if they both get taken down, that might actually create a viable lane for a faux-“reasonable” GOP presidential candidate to win the nomination in 2024 who might be a stiffer challenge to beat, with the media fawning on their alledged moderation and beating the Biden fatigue/age drumset.
David Anderson
@cmorenc: Kemp and Sununu are playing for that outcome,
Tenar Arha
@David Anderson: The Kentucky Medicaid expansion vs. Tennessee’s refusing the same, well those comparisons haunt me. I think that’s when I (layperson) learned the term excess deaths.
WeimarGerman
If you’re interested, ISPOR had an issue on real world evidence a few years ago. Here’s an interview with some of the authors.
David Anderson
@WeimarGerman: i’ve worked with Nirosha at Duke Margolis…. Very good mind
TF79
As far as the general point goes regarding inference and natural experiments goes, this seems spot on (we don’t always need RCTs to learn things). But in this particular case, what would make the timing of treatment a source of as-good-as-random variation here that we can think of this as a natural experiment? It seems like lots of potential time -varying unobservables could drive both timing and outcomes in confounding ways. AEA P&P is not peer reviewed, but I imagine if this was submitted to a peer reviewed journal, referees would ask some hard questions along these lines. That said, this seems a substantial improvement over prior work in terms of getting a sense of treatment efficacy, though perhaps not airtight enough with identification to convince those who were already skeptical.
Unkown known
Did you know that everyone talks like heart attacks are bad for you, but we have absolutely no double blind randomized controlled trials for this? We’ve literally never done the study where a bunch of people have a heart attack induced, to see if they fare any worse than people who were just given a placebo electric shock.
I demand that hospitals stop wasting all that money on those big expensive cardiac units, until we have proper evidence to base our decisions on.
(I’m out of date on what the irony font is)