Re-Analysis, Reversal and Reflection

Getting acute ischemic stroke treatment right is better than doing it wrong.

The use of alteplase for acute ischemic stroke has been described as the biggest, baddest controversy in emergency medicine.[¹]  I have published and spoken widely on the subject and describe it as the gift that keeps on giving. Some people argue that the heat has been taken out of the issue, due to the advent of endovascular clot retrieval, which has become the high-profile approach for acute ischemic stroke and the role of stroke centers.

However, stroke thrombolysis is a living example of persistent problems with methodology, lack of critical appraisal skills, bias, conflicts of interest and clinical guidelines. These contributors to how we can be fooled by the literature are constantly present in medicine. This is currently explicit with the pandemic and the use of preprint archives to promote studies prior to peer reviewed publication. History may view this as causing more harm than benefit due to the premature adoption of therapies.

ADVERTISEMENT

Background

This paper has been stimulated by a re-analysis of ECASS-III, but it is first worthwhile to outline the process of how dogma is created (adapted from Farkas)[²]:

1. Clinical trials test a new intervention. The trials are designed to include people who are most likely to benefit, and exclude people most likely to be harmed. This goes to efficacy: who benefits in the highly controlled setting of a clinical trial.
2. A signal of benefit is found and use is promoted. For expensive interventions, there are also commercial interests at play. Nevertheless, initial use is typically somewhat cautious (not always) and limited to those who most resembled those in the clinical trials.
3. Over time, those who were excluded from the trials are potentially missing out on the intervention, leading to indication creep. This is exacerbated by the human brain that is hard wired for simplicity. Ultimately, there is limited incentive to definitively evaluate the spectrum of who should and should not receive the intervention.
4. The intervention becomes a widely accepted standard of care and the details of the original studies are forgotten. The indications to use the intervention become more inclusive, including people who were originally excluded.
5. Leaders opine that further evaluation of the intervention, especially in those who were originally excluded, would be unethical. The classic example of this is the Cardiac Arrhythmia Suppression Trial (CAST) which tested whether prophylactic anti-arrhythmics improved outcomes after myocardial infarction (MI).[³] Cardiologists were so confident in this management, that recruitment to CAST was slow, as many felt that it was unethical to allow patients to receive placebo after MI.[⁴] CAST reached the opposite conclusion to the one expected and showed that the best mechanistic reasoning can be wrong.[⁴]  This illustrates how once the dogma is established and both patients and the system expect to adopt the intervention, it requires bravery to test it in a rigorous fashion.

One of the negative consequences of the premature adoption of novel therapies is that subsequent research assumes its value and enhances it with faster and wider implementation. This has led to the phenomenon of medical reversal.

ADVERTISEMENT

Run it back

Medical reversal occurs when current practice is found to be no better than placebo (or its omission) in well done clinical trials.[⁴] Prasad’s work on medical reversal shows that when practices — supported by inferior evidence — are retested in powerful randomized trials, nearly half of them fail. While there is some physiologic plausibility for thrombolysis in stroke, it is worth remembering that every clinical trial that has failed had a sound physiologic basis.

Is medical reversal brewing for stroke thrombolysis? Many doctors are simply unaware that of the 12 stroke thrombolysis trials that led to its use, only two report a statistical benefit on their primary outcome (NINDS and ECASS-III) that were foundational for current practice. Of the 10 that failed to find benefit for stroke thrombolysis, four had to be stopped early for harm (increased mortality).

ADVERTISEMENT

ECASS III (European Cooperative Acute Stroke Study, n = 821) was published in 2008 and assessed outcomes of acute ischemic stroke using tPA vs. placebo at 3-4.5 hours after stroke onset.[⁵]

They reported a seven percent absolute benefit of improved modified Rankin Score (mRS) 0-1 at 90 days compared to control, nine percent increase in intracranial hemorrhage, two percent increase in symptomatic intracranial hemorrhage and no significant difference in mortality. This year, a re-analysis of ECASS III was published which concludes: “Reanalysis of the ECASS III trial data with multiple approaches adjusting for baseline imbalances does not support any significant benefits and continues to support harms for the use of alteplase 3–4.5 hours after stroke onset. Clinicians, patients and policymakers should reconsider interpretations and decisions regarding management of acute ischaemic stroke that were based on ECASS III results.”[⁶]

This finding emphasises that a clinical trial has internal validity if and only if the imbalance between groups, bias in the assessment of outcome and chance, have been excluded as possible explanations for the difference in outcomes. This was highlighted by Shy in 2014, who noted that the online version of the ECASS III paper was changed in 2013 to reflect the actual p value of 0.003, vs. the originally published p 0.03.[⁷]

The authors originally defined significant p values for these comparisons as < 0.004, so the correction marked stroke history as a significant difference between the two groups. Clinically, it is known that recurrent strokes have a worse outcome than first strokes. So, the difference in outcome could be fully explained by the baseline imbalance.

ADVERTISEMENT

Balancing the baseline

Baseline imbalance in stroke severity is a recurring theme. The initial trial of Factor VII for intracerebral hemorrhage was ‘positive,’ but had a baseline imbalance in stroke severity: the placebo group had more severe strokes.[⁸] When this trial was repeated with a larger sample size that achieved balanced severity between the groups, there was no patient oriented advantage for Factor VII.[⁹]

The original study that set us on this course was NINDS (National Institute of Neurologic Disorders and Stroke, n = 624), published in 1995.[¹⁰] This study reported that tPA, given within three hours of stroke onset, led to improved outcomes at three months (mRS 0-1 39% tPA vs. 26% placebo). The 2009 reanalysis found that the baseline imbalance in stroke severity where the placebo group had more severe strokes, led to the difference in outcomes.[¹¹]

If tPA really works, we should see a bigger change in the NIHSS (National Institutes of Health Stroke Severity) score in the tPA group vs. the placebo group. Yet the difference was 0.0.

IST-3 (International Stroke Trial, n = 3035) was the largest clinical trial looking at tPA for acute ischemic stroke up to six hours after onset.[¹²] It failed to show a benefit on its primary outcome. In the 3-4.5 hour subgroup (n = 1177), no difference in functional outcomes in those randomized to tPA vs. placebo (32% tPA vs. 38% placebo (OR 0.73) was reported. However, this was obtained by using a 99% confidence interval (0.50-1.07). Readers will be aware that the conventional reporting approach is the 95% CI. Using the traditional 95% CI results in a statistically significant association with placebo for good functional outcome compared to tPA.[¹³]

Even without re-analysis, there is fragility in the data, which can be quantified using the fragility index (FI). The FI is a method to understand if a study is statistically reproducible. It is the minimum number of patients who would need to have a different outcome to change the p value from < 0.05 to > 0.05, i.e. from statistically significant to insignificant.

It is calculated by repeatedly applying Fisher’s exact test, while successively reallocating patients, one at a time, from the favourable outcome group to the control group. The FI of the NINDS trial is three. This means that only three patients would have to have had a different outcome to change the study from positive to negative. The FI of ECASS-III is one. It defies credibility to understand how these fragile studies had the effect of changing clinical practice.[¹⁴]

Going forward

The two re-analyses of the only two foundational studies that were used to adopt thrombolysis support this becoming another medical reversal. There are now zero foundational studies supporting this practice.

In recent times there has been a strong focus on knowledge translation and implementation science. It is widely considered that the average time for knowledge translation for new findings to be adopted into practice is 17 years.[¹⁵] Online FOAMED sources promote aims of reducing this to one year.

However, people rarely consider the converse: how long does it take, on average, to reverse an erroneous practice? Ironically, it was the neurologist John Hughlings Jackson who is quoted to have said: It takes 50 years to get a wrong idea out of medicine and 100 years to get a right one into medicine. Perhaps getting the wrong idea out of medicine only takes 50 years. If the story of stroke thrombolysis is one day used as a medical reversal, then we’re only half way there.

Can’t we do better?

REFERENCES

1. SoRelle R. The ‘Biggest, Baddest’ Controversy in EM. Emergency Medicine News 2013; 35(4): 1.
2. Farkas J. PulmCrit- Dogmalysis of PCI for NSTEMI patients with a history of CABG2017. http://emcrit.org/pulmcrit/pci-nstemi-cabg/ (accessed.
3. Echt DS, Liebson PR, Mitchell LB, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med 1991; 324(12): 781-8.
4. Prasad V, Cifu A. The reversal of cardiology practices: interventions that were tried in vain. Cardiovascular diagnosis and therapy 2013; 3(4): 228-35.
5. Hacke W, Kaste M, Bluhmki E, et al. Thrombolysis with alteplase 3 to 4.5 hours after acute ischemic stroke. N Engl J Med 2008; 359(13): 1317-29.
6. Alper BS, Foster G, Thabane L, Rae-Grant A, Malone-Moses M, Manheimer E. Thrombolysis with alteplase 3-4.5 hours after acute ischaemic stroke: trial reanalysis adjusted for baseline imbalances. BMJ Evid Based Med 2020.
7. Shy BD. Implications of ECASS III error on emergency department treatment of ischemic stroke. The Journal of emergency medicine 2014; 46(3): 385-6.
8. Mayer SA, Brun NC, Begtrup K, et al. Recombinant activated factor VII for acute intracerebral hemorrhage. N Engl J Med 2005; 352(8): 777-85.
9. Mayer SA, Brun NC, Begtrup K, et al. Efficacy and safety of recombinant activated factor VII for acute intracerebral hemorrhage. N Engl J Med 2008; 358(20): 2127-37.
10. Tissue plasminogen activator for acute ischemic stroke. The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. N Engl J Med 1995; 333(24): 1581-7.
11. Hoffman JR, Schriger DL. A graphic reanalysis of the NINDS Trial. Ann Emerg Med 2009; 54(3): 329-36, 36 e1-35.
12. Sandercock P, Wardlaw JM, Lindley RI, et al. The benefits and harms of intravenous thrombolysis with recombinant tissue plasminogen activator within 6 h of acute ischaemic stroke (the third international stroke trial [IST-3]): a randomised controlled trial. Lancet 2012; 379(9834): 2352-63.
13. Brown MD, Burton JH, Nazarian DJ, Promes SB. Clinical Policy: Use of Intravenous Tissue Plasminogen Activator for the Management of Acute Ischemic Stroke in the Emergency Department. Ann Emerg Med 2015; 66(3): 322-33.e31.
14. Fatovich DM. The “Fragility” of Stroke Thrombolysis. Tasman Medical Journal 2020; 2(1): 6-10.

15. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. Journal of the Royal Society of Medicine 2011; 104(12): 510-20.