I’ve been conferencing up and down the Eastern seaboard for the past week and many of my colleagues and collaborators are at their specialty major conferences as well. I found this from the American Society of Clinical Oncologists (ASCO) to be fascinating.
Real-world patients *do not* do as well as those enrolled in clinical trials.
Very cool analysis by Angela Green, @peterbachmd, Sham Mailankody &co @ASCO #ASC019 @carriebennette pic.twitter.com/hd3kHMfFB7— Aaron Mitchell (@TheWonkologist) June 1, 2019
This study looks at the survival improvement results reported from the Phase 3 clinical trials of major (and expensive) new cancer drugs and then compares the results from a major cancer registry.
Ideally, the results from the clinical trial are the same as the real world results or perhaps slightly worse as new technologies, techniques, and learning by doing will have occurred to get a little bit of an incremental improvement wedge. That is not the usual case. Instead, the real world results are a bit worse than the clinical trial results.
Why does this matter beyond the obvious that lower survival times are less desirable than longer survival times?
As we move towards value based and outcome based contracts, we need to figure out which evidence is reliable and what things we consider in the creation of the contracts. The trial evidence is scientifically rigorous but for some reason, the translation suffers some degradation. Should initial contract phases be based on unadjusted clinical trial outcomes? Should there be a discount rate with some type of upside kicker to account for the likely case of lower pragmatic performance while adding some space for gains in the case of a happy surprise? How long should contracts run until they are renegotiated or re-benchmarked against pragmatic, real world evidence?
I don’t know the answers to any of those questions, but this study raises these thorny implications.
daveNYC
A lot of those data points look like the real world results are a bit more than ‘a bit’ worse than the clinical trials. I’m eyeballing, but a lot of them seem to be clocking in with the clinical trials having at least 50% more survival time than ‘da world. Maybe not the biggest thing for Erlotinib as far as raw numbers goes, but pretty damn huge for Everolimus.
I assume that they’re comparing apples and apples with the specifics of each patient? I imagine that the clinical trials for cancer drug X are looking for people who have cancer X and nothing else, while Medicare is treating everyone no matter what other conditions they have in addition to the cancer.
Barry
I’m not surprised; clinical trial patients will be a screened group, and should have the lower tail of outcomes trimmed.
Brachiator
I’m not sure that the chart tells you why there is a difference.
And are the same drugs and treatment protocols involved?
Argiope
I have questions. Do we have data that suggests the “other conditions” of patients in the real world are primarily responsible for shorter survival? Are the follow-up protocols potentially involved, and if so, what lessons are there for our patient support structures in the real world? What would the healthcare system need to do to treat every patient like a study participant, and how resource-intensive (and feasible) would that be? Is it possible to capture the qualities that make an individual more likely to say yes to inclusion in a research trial? Are some of those qualities amenable to policy changes, e.g. socioeconomic status?
Thanks, David, for this interesting (and sobering) post.
JCJ
@daveNYC:
This reminds me of a statement made by a thoracic surgeon years ago regarding surgery (extrapleural pneumonectomy) for mesothelioma. Someone cited the results reported by Dr Sugarbaker at Harvard. The surgeon pointed that many of the patients in that report had to be well enough to fly to Boston to have the procedure done compared with the patient being presented who was not in as good of condition.
It would be very important to be certain that the patients were comparable (multiple sites of metastatic disease vs few), overall condition (performance status), age, etc.
A recent study reported on men with metastatic prostate cancer who uderwent radiation to the prostate gland. That has not been done in the past, but this study showed local treatment was beneficial in men with oligometastatic disease vs no benefit in widely metastatic disease.
daveNYC
@Brachiator: The drugs should be the same, otherwise there’s no way for it to make any sense. The treatment protocols may be different, but I don’t think there’s details on if they are or not.
Has anyone floated the idea that the clinical trials’ numbers are cooked? The difference between the trial and real-world survival times for Everolimus are absolutely huge. To the point where I’m wondering if the SEER-Medicare patients are getting half-doses or just Flintstones chewables and Zima.
Anonymous At Work
Doctors who view themselves as “artists” (I have heard some proclaim such) then “re-translate” the trial evidence of best practices into what they would consider best, based on anecdotal history. Really, they are taking their medical professors’ word on what is best, based on what those medical professors learned when they were in medical school.
Brachiator
@daveNYC:
A recent episode of the BBC program “More or Less” comes to mind.
A doctor had murdered over 200 patients and investigators asked if statistics might have helped catch him earlier.
The answer was Yes, but you had to collect and analyze the right data.
Here we are not looking for deaths, but the reasons for different outcomes. This chart is just the beginning.
OzarkHillbilly
Relevant? Use of male mice skews drug research against women, study finds
Male animal bias is unjustified and can lead to drugs that work less well for women
Dr Ronnie James DO
@Anonymous At Work: Ideally, doctors are supposed to be trained to “become their own professors” / self-improving experts: keeping up on the literature and trial outcomes, assessing it critically, making their own judgements and applying their best judgment to the patient in front of them. This is an increasing part of medical school curricula; however, you are right that many physicians will still take what they learned in med school / residency, then stop studying, filter it through their personal “anecdata,” and call it a day.
former doc
Clinical trials have eligibility criteria to try to ensure uniformity of patients since trials are conducted in many institutions and even multinationally. Further, it is cumbersome to enroll people. As a result, trials have the healthy end of the disease spectrum.
Barry
@Brachiator: “I’m not sure that the chart tells you why there is a difference.
And are the same drugs and treatment protocols involved?”
These sorts of comparisons are mature by now (see ‘propensity score’ for a start).
MattF
Unfortunately, the fact that the companies conducting clinical trials are hoping for success will bias the results. These biasses can be very subtle or very obvious, but they are unavoidable.
One signal for bias is whether a strong result in a clinical trial is correlated with a similarly strong result in-real-life. Might be worthwhile to look at that.
Anonymous At Work
@Dr Ronnie James DO: This point is somewhat related to the protester grabbing the mic from Senator Harris this weekend. “The unearned confidence of middle-aged white men.”
Barbara
@Dr Ronnie James DO: I would say that professional stasis is less likely to be the case in oncology than in nearly any other medical specialty. Surgical specialties are probably more hidebound. I assume that differences in real world results are most likely a function of wider range of use ( e.g., starting an agent later in the progression of the disease) and wider range of patients (older, sicker) and less support (side effects less well managed).
Barbara
@Barry: I suppose there are doctors who design their own protocols, but given the toxicity of cancer drugs, I think most follow standardized protocols if they exist.
Doug R
I see two trendlines, the line drawn seems to be a crappy median of two obvious trends. That’s the point?
Dr Ronnie James DO
@Barbara: Based on my own personal exposure (NB I’m a family med doc and former clinical cancer researcher) I…have seen little to no evidence to disprove your assertion. The incredible volume of clinical cancer research helps drive Hem/Onc forward; whereas so much of progress in surgery appears to be industry driven (staplers! robots!)
bluefoot
@Argiope: I am currently working on a project analyzing this, but for a different area than oncology. There are two big differences between clinical trials and real world. One, a clinical trial has specific inclusion and exclusion criteria. When you’re trying to figure out of a new drug works, you try to minimize things that might confuse the data. So patients with some co-morbidities might be excluded (for example), or you have a restricted age range, or patients on certain other drugs may be excluded, etc. Or the patients for the clinical trial are in a certain stage of the disease.
Two, care in a clinical trial (at least for the disease I work in) is a lot more consistent – mandated by the protocol – than what happens in real world. All patients get the drug on schedule, get clinical assessments and testing every x weeks, an MRI every y months (or more if needed). In the real world, your insurance company might only approve an MRI once a year (or less) so your disease isn’t monitored to the same extent. A patient may not get or take their medication on as strict a schedule. You might not be able to get an appointment with a specialist in a timely way if your condition changes or worsens, etc.
Also, there are many diseases where “standard of care” isn’t really well established, so there’s a lot of variability from clinician to clinician in how they treat. There can even be a lot of variability in how they assess a patient, and consequently treatment.
Redshift
I recall reading an article a while back about an as yet unexplained effect that patients (in general) who got more personal attention from doctors did better. Things like a sympathetic pat on the hand, more eye contact, more time just talking and listening.
I would imagine patients in clinical trials get a lot more personal attention than patients in regular treatment.
(And of course this is something our health care system discourages doctors from doing…)
psycholinguist
doesn’t have to be one thing, could be several factors that all tend to push down the efficacy rate a bit. As people have already implied:
1. greater variability in quality of the Doctor and her or his protocol
2. greater variability in time of adoption, and that’s going to be a skewed distribution towards sicker people (e.g. you don’t start taking a cancer drug as a prophylactic)
3. Patients often screw around with protocols. Might cut the pill in half because they can’t afford, think they know better than the doctor, may be taking other stuff and don’t tell the doc, etc. That’s going to be more tightly controlled in a clinical trial
4. regression to the mean? Psychology is dealing with a version of this right now with the recent replication of studies industry.
StringOnAStick
A friend of ours, who has since passed, was unable to get into any clinical trials for his cancer because his weight was already too low. So, some of what you are seeing in the data may have a lot to do with the fact that you have to be in better shape to qualify for a clinical trial. The canard that the only people they let into these “experimental” treatments are those who have no options left fails to note that being too far along in the course of your disease keeps you out of the trials.
I participated in a clinical trial for an artificial meniscus that was supposed to last 10 to 15 years; it’s year 3, it is breaking down and I’m back in pain so I’m bailing on the trial and getting that knee replaced in July. The trial doctor wants to put another one in there but I’m not doing open joint surgery every 3 years and going through rehab that isn’t much different from what I’ve recently dealt with by getting my worse knee replaced. Note that my “worse” knee didn’t qualify for the clinical trial because it had OA that was too severe, an interesting parallel to the clinical trials for drugs you are discussing here.
scav
It is interesting to think of, and all its implications. Shouldn’t really have been surprising though. Reality gets so very tidied away to become a mathematical model, but also for a clinical trial. I can see how we’d design trials to maybe deal with the selection of patients (to a certain degree) but run into issues with testing for the variablity of medical contexts — especially with how to design a blind test, plus the whole ethical thing of being aware of less than best preactices and letting them continue).
texasdoc
@StringOnAStick: You make a very valid point, that those in a trial are selected and don’t represent the full array of patients. In oncology, we joke that patients are trials are healthy, they just happen to have cancer. As a result, you have to take trial results with a large grain of salt. In the real world, we treat patients with all kinds of comorbidities which can limit treatment options/effectiveness. And there are some patients we can’t treat at all. I remember when I was in training, analyzing the response/survival of the acute myelogenous leukemia patients we had treated and finding a much lower complete response rate than in clinical trials, so this is something that has been known forever. It should also slow any efforts to tie reimbursement to treatment results, without the kind of granular data that would allow you to predict results for individual patients. Most doctors don’t see enough patients with a particular kind of cancer at identical stages to have significantly valid results.
texasdoc
@StringOnAStick: You make a very valid point, that those in a trial are selected and don’t represent the full array of patients. In oncology, we joke that patients are trials are healthy, they just happen to have cancer. As a result, you have to take trial results with a large grain of salt. In the real world, we treat patients with all kinds of comorbidities which can limit treatment options/effectiveness. And there are some patients we can’t treat at all. I remember when I was in training, analyzing the response/survival of the acute myelogenous leukemia patients we had treated and finding a much lower complete response rate than in clinical trials, so this is something that has been known forever. It should also slow any efforts to tie reimbursement to treatment results, without the kind of granular data that would allow you to predict results for individual patients. Most doctors don’t see enough patients with a particular kind of cancer at identical stages to have statistically valid results.
Fair Economist
Not to disagree with any of the real-world reasons trial patients may actually be doing better, but don’t discount the possibility the trial results could be wrong. There’s the “file drawer” effect – some trials of useless interventions will show a positive result by statistical accident (19 go to the file drawer, one to the FDA). There are a lot of trials, so there’s going to be a nontrivial rate of these bogus results. This is sometimes amplified by researchers gaming in-trial dropout rules or by doing reanalysis (every reanalysis is another chance for a random bogus positive), although people are more aware of those these days so it’s harder. In addition, there’s a lot of money in clinical trials and the possibility of intentional fraud is there too.
Disclosure – I was involved in clinical trial research in the 90’s and I saw a lot of self-delusion. I don’t think there was any fraud, but after working years on developing a treatment people become absolutely convinced it’s going to work. If the results were negative, they were *sure* the results were wrong somehow. TBH, I got it too sometimes.
anon
Here’s the weird thing. For eve-Breast, the survival rate shown is not just lower in the real world than the clinical trial, it’s lower than the non-intervention arm of the clinical trial. I don’t know about the others, but that is startiling.
J R in WV
My father had CMML leukemia, a rare (one in 450,000 cases of leukemia) disorder. He was treated at M D Andersen center in Houston with harsh chemotherapy while spending 30 days in a clean room, isolated from nature while his immune system was destroyed.
This was done so that he might qualify for a clinical trial. He was the first CMML patient to present and they wanted to see if their new drug would work on it at all. It did work, but the original chemo done to qualify him for the trial caused so much lung damage that the resultant COPD was eventually fatal.
I’m not sure if he made 5 years or not, that wasn’t important to me. Hospice was a real blessing to him… he died on election day, 2004, which was a really bad day for us. Fortunately my RWNJ brother didn’t speak of the election ever. Dad did get to spend time with his grandkids, which was a blessing for him. I don’t know if the kids really benefited as he was pretty much an invalid in their memories.
It is obvious to me from our experience that clinical trials select patients with great care and after much examination and sometimes treatment that would have been prohibitively expensive for the average patient. On the other hand, we did get a lot of time with my dad, and I imagine the drug is now available to patients in places other than M D Anderson. White powder in anonymous clear gel caps. No idea of the name, just a number assigned. Leaping to the conclusion he was receiving the drug rather than in a control group. The chemo would have been brutal to administer to a control group patient, plus their only shot at a patient with CMML gone acute.
I also have a dear friend who manages the execution of clinical trials at various locations all over the world, trying to make sure doctors in Brazil and the EU are using exactly the same treatment protocols as designed by her firm and their big pharma clients. Stressful work for sure.
TL;DR
Clinical trials are extremely selective about who they admit to their testing program. It is no surprise that results from such a trial is far better than results in normal medical practice.
Ruckus
@Dr Ronnie James DO:
I’ve met a couple of those. The first one was all knowing. His experience was his guide, not that of the wider medical world. The second just doesn’t seem to give a damn unless it was something unusual he was looking at. The first, I think age was an issue, he’d been practicing for a long time, I’d guess about 40 yrs. The second, middle aged, I think was either bored or overwhelmed beyond his abilities.
All but one of the younger docs I’ve seen, and I see a lot of residents, are far more conscientious to their patients actual well being, rather than the docs expectations. The older, set in their ways docs, in my opinion, seem to believe that they have the answers and expect X outcome from their treatment. When the treatment results fall far outside their expectations, the reason falls upon the patient rather than the treatment, and no adjustments on their side are required. And this is not at all to say that age is the problem, I’ve had several older docs who are very open to differing outcomes and treatment options. I believe it is a medical system issue though because it seems to happen often enough to be rather noticeable.
jl
IMHO, access to care sucks in the US, even for Medicare patients. ‘Standard of care’ is a comforting fiction for the average patient in the US, and very different than standard of care in a clinical trial. Looking at international comparisons, US medical care for the average patient, no matter what the program, private or public, is compromised due to outrageous corporate price gouging and sloppy safety practices. What happens when short run corporate profits are everything, and in a ruthless under regulated market in which non-profits are forced to act very much like ruthless money grubbing for profit organizations.
But, that is just an ‘IMHO;.
JGabriel
David Anderson @ Top:
Of course it isn’t.
Clinical trials are typically done by people selected (or self-selected) for their carefulness in following protocols, they are frequently trained in the protocols they are supposed to follow for a particular drug trial, and the patients are frequently weeded out for any co-factors that might complicate the results – none of which is necessarily true in the real world.
The results from the trials, particularly late stage trials, should be expected to typically be ideal case scenarios. I’m surprised anyone is surprised by this.
But I’m not a clinical scientist (although I did do IT for a clinical science team at a major drug company for a few years), so maybe my view is misinformed.
The Castle
It has been recognized for a while now that there is a replicability crisis in many areas of science. Oncology medicine is one of the worst offenders.
Reasons for this crisis probably include some of the things other jackals have mentioned. There are others.
As a research scientist, I can tell you that a big structural/cultural problem is the enormous amount of money being put into medicine and the parallel expectation of miracles, fast. Good science is usually somewhat slow and methodical, but that is very hard to do in the current climate. Worse, something like cancer is really hundreds of separate diseases/causes lumped under the same label, which makes it an especially difficult subject to study.
An excellent scientist will bend over backwards to try to think of (and rule out) alternate explanations for a study result.
Science is a philosophical system I like because it can help keep us from fooling ourselves. There is much power in this. But there is zero support for doing this kind of reflection. There is so much pressure to Do Something. And the intense pressure to Publish or Perish and Find Cures Now leads to the current state of affairs, where much of the scientific literature is not particularly trustworthy.
Barbara
A real problem here is that, categorically, the FDA has a low bar for demonstrating efficacy for many new drugs, but especially oncology drugs. I think that it should be assumed that the clinical trial will show the best results possible, and that “real world” results will be worse for all the reasons that have been mentioned.
Barry
@Barbara: “I suppose there are doctors who design their own protocols, but given the toxicity of cancer drugs, I think most follow standardized protocols if they exist.”
I was referring to statistical analysis protocols.
Barry
@Doug R: “I see two trendlines, the line drawn seems to be a crappy median of two obvious trends. That’s the point?”
The line is the famous ‘Y=X’ line beloved of equivalence analyses all over the world :)