Outcome switching
Outcome switching is the practice of changing the primary or secondary outcomes of a clinical trial after its initiation. An outcome is the goal of the clinical trial, such as survival after five years for cancer treatment. Outcome switching can lead to bias and undermine the reliability of the trial, for instance when outcomes are switched after researchers already have access to trial data. That way, researchers can cherry pick an outcome which is statistically significant.
Problem
[edit]Outcome switching can undermine the reliability of the trial, for instance when outcomes are switched when researchers already have access to the trial data. It can lead to bias in terms of benefits and harms. For example, when the findings using the original protocol were statistically insignificant, a study may cherry pick a new outcome measure that is statistically significant.[1] If there are a large number of outcomes to choose from, it is likely at least one will show significant findings, assuming no correction is made for multiple testing.[2] Incomplete or misleading reporting of outcomes is an example of unethical research practice.[1]
Possible solutions
[edit]Primary outcomes for clinical trials need to be defined upfront to prevent a biased selection of outcomes. Some medical journals require adherence to the CONSORT standards, which require authors to document any deviations from the initial study protocol and explain why changes were made. For instance, JAMA and the British Medical Journal require inclusion of the CONSORT checklist.[1] When pre-specified outcomes are defined but unclear, researchers still have wriggle room to choose the most favourable option. A clear pre-defined outcome is therefore important to avoid outcome switching.[3]
Study authors may engage in outcome switching due to high pressures to publish. It is more difficult to publish negative results, especially in high-impact journals. Registered reports—a type of journal article where editors pledge to publish clinical trials irrespective of their results—may help combat this pressure to produce positive results. Initial peer review takes place based on the methodology and the reasoning behind the study.[4]
Prevalence
[edit]Despite problems with outcome switching, the practice is common. Changes in primary outcome metrics are present in nearly one in three studies.[5] Outcome switching also occurs frequently in follow-up studies.[6] In an analysis of oncology trials, outcome switching was more common in studies with a male first author, and in studies funded by non-profits.[7] One study analysed outcome switching in five top medical journals, writing letters for each misreported trial outcome. Journal editors and clinical trial authors typically responded dismissively when concerns were raised, misrepresenting ethical standards and including ad hominem attacks.[8]
Examples
[edit]A drug against major depressive disorder, paroxetine, was marketed for children and teenagers for years, based on a flawed trial that involved outcome switching.[9] The trial's protocol had described two primary and six secondary outcomes by which it would measure efficacy.[10][11] The data showed that, according to those eight outcomes, paroxetine was no more effective than placebo. According to Melanie Newman, writing for the BMJ, "[t]he drug only produced a positive result when four new secondary outcome measures, which were introduced following the initial data analysis, were used instead. Fifteen other new secondary outcome measures failed to throw up positive results."[a][11]
See also
[edit]References
[edit]- ^ Melanie Newman, BMJ, 2010: "Study 329's results showed that paroxetine was no more effective than the placebo according to measurements of eight outcomes specified by Martin Keller, professor of psychiatry at Brown University, when he first drew up the trial.
"Two of these were primary outcomes: the change in total Hamilton Rating Scale (HAM-D) score, and the proportion of 'responders' at the end of the eight-week acute treatment phase (those with a ≥50% reduction in HAM-D or a HAM-D score ≤8). The drug also showed no significant effect for the initial six secondary outcome measures.
"The drug only produced a positive result when four new secondary outcome measures, which were introduced following the initial data analysis, were used instead. Fifteen other new secondary outcome measures failed to throw up positive results.".[12]
- ^ a b c Altman DG, Moher D, Schulz KF (February 2017). "Harms of outcome switching in reports of randomised trials: CONSORT perspective". BMJ (Clinical Research Ed.). 356: j396. doi:10.1136/bmj.j396. PMID 28196813. S2CID 206916748.
- ^ Wicherts JM (2017). "The Weak Spots in Contemporary Science (and How to Fix Them)". Animals. 7 (12): 90. doi:10.3390/ani7120090. ISSN 2076-2615. PMC 5742784. PMID 29186879.
- ^ Kahan BC, Jairath V (2018-05-02). "Outcome pre-specification requires sufficient detail to guard against outcome switching in clinical trials: a case study". Trials. 19 (1): 265. doi:10.1186/s13063-018-2654-z. ISSN 1745-6215. PMC 5932799. PMID 29720248.
- ^ Chambers C (2017-08-24). "Clinical trials revolution could change the future of medical research". The Guardian. ISSN 0261-3077. Retrieved 2024-02-19.
- ^ Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF (November 2015). "Comparison of registered and published outcomes in randomized controlled trials: a systematic review". BMC Medicine. 13: 282. doi:10.1186/s12916-015-0520-3. PMC 4650202. PMID 26581191.
- ^ Kampman JM, Sperna Weiland NH, Hollmann MW, Repping S, Hermanides J (September 2021). "High incidence of outcome switching observed in follow-up publications of randomized controlled trials: Meta-research study". Journal of Clinical Epidemiology. 137: 236–240. doi:10.1016/j.jclinepi.2021.05.003. PMID 34004339.
- ^ Falk Delgado A, Falk Delgado A (August 2017). "Outcome switching in randomized controlled oncology trials reporting on surrogate endpoints: a cross-sectional analysis". Scientific Reports. 7 (1): 9206. Bibcode:2017NatSR...7.9206F. doi:10.1038/s41598-017-09553-y. PMC 5569019. PMID 28835682.
- ^ Goldacre B, Drysdale H, Marston C, Mahtani KR, Dale A, Milosevic I, Slade E, Hartley P, Heneghan C (February 2019). "COMPare: Qualitative analysis of researchers' responses to critical correspondence on a cohort of 58 misreported trials". Trials. 20 (1): 124. doi:10.1186/s13063-019-3172-3. PMC 6374909. PMID 30760328.
- ^ Belluz J (2015-12-29). "How researchers dupe the public with a sneaky practice called "outcome switching"". Vox. Retrieved 2024-03-18.
- ^ "Study drug: BRL29060/Paroxetine (Paxil)", SmithKline Beecham, 20 August 1993, amended 24 March 1994.
- ^ a b Jureidini, Jon N.; McHenry, Leemon B.; Mansfield, Peter R. (2008). "Clinical trials and drug promotion: Selective reporting of study 329", International Journal of Risk & Safety in Medicine, 20(1–2) (pp. 73–81), p. 74. doi:10.3233/JRS-2008-0426
- ^ Newman, Melanie (11 December 2010). "The Rules of Retraction", BMJ, 341(7785), pp. 1246–1248. doi:10.1136/bmj.c6985 PMID 21138994 JSTOR 20800711