Study heterogeneity

In statistics, (between-) study heterogeneity is a phenomenon that commonly occurs when attempting to undertake a meta-analysis. In a simplistic scenario, studies whose results are to be combined in the meta-analysis would all be undertaken in the same way and to the same experimental protocols. Differences between outcomes would only be due to measurement error (and studies would hence be homogeneous). Study heterogeneity denotes the variability in outcomes that goes beyond what would be expected (or could be explained) due to measurement error alone.^[1]

Introduction

Meta-analysis is a method used to combine the results of different trials in order to obtain a quantitative synthesis. The size of individual clinical trials is often too small to detect treatment effects reliably. Meta-analysis increases the power of statistical analyses by pooling the results of all available trials.

As one tries to use meta-analysis to estimate a combined effect from a group of similar studies, the effects found in the individual studies need to be similar enough that one can be confident that a combined estimate will be a meaningful description of the set of studies. However, the individual estimates of treatment effect will vary by chance; some variation is expected due to observational error. Any excess variation (whether it is apparent or detectable or not) is called (statistical) heterogeneity.^[2] The presence of some heterogeneity is not unusual, e.g., analogous effects are also commonly encountered even within studies, in multicenter trials (between-center heterogeneity).

Reasons for the additional variability are usually differences in the studies themselves, the investigated populations, treatment schedules, endpoint definitions, or other circumstances ("clinical diversity"), or the way data were analyzed, what models were employed, or whether estimates have been adjusted in some way ("methodological diversity").^[1] Different types of effect measures (e.g., odds ratio vs. relative risk) may also be more or less susceptible to heterogeneity.^[3]

Modeling

In case the origin of heterogeneity can be identified and may be attributed to certain study features, the analysis may be stratified (by considering subgroups of studies, which would then hopefully be more homogeneous), or by extending the analysis to a meta-regression, accounting for (continuous or categorical) moderator variables. Unfortunately, literature-based meta-analysis may often not allow for gathering data on all (potentially) relevant moderators.^[4]

In addition, heterogeneity is usually accommodated by using a random effects model, in which the heterogeneity then constitutes a variance component.^[5] The model represents the lack of knowledge about why treatment effects may differ by treating the (potential) differences as unknowns. The centre of this symmetric distribution describes the average of the effects, while its width describes the degree of heterogeneity. The obvious and conventional choice of distribution is a normal distribution. It is difficult to establish the validity of any distributional assumption, and this is a common criticism of random effects meta-analyses. However, variations of the exact distributional form may not make much of a difference,^[6] and simulations have shown that methods are relatively robust even under extreme distributional assumptions, both in estimating heterogeneity,^[7] and calculating an overall effect size.^[8]

Inclusion of a random effect to the model has the effect of making the inferences (in a sense) more conservative or cautious, as a (non-zero) heterogeneity will lead to greater uncertainty (and avoid overconfidence) in the estimation of overall effects. In the special case of a zero heterogeneity variance, the random-effects model again reduces to the special case of the common-effect model.^[9]

Common meta-analysis models, however, should, of course, not be applied blindly or naively to collected sets of estimates. In case the results to be amalgamated differ substantially (in their contexts or in their estimated effects), a derived meta-analytic average may eventually not correspond to a reasonable estimand.^[10]^[11] When individual studies exhibit conflicting results, there likely are some reasons why the results differ; for instance, two subpopulations may experience different pharmacokinetic pathways.^[12] In such a scenario, it would be important to both know and consider relevant covariables in an analysis.

Testing

Statistical testing for a non-zero heterogeneity variance is often done based on Cochran's Q^[13] or related test procedures. This common procedure however is questionable for several reasons, namely, the low power of such tests^[14] especially in the very common case of only few estimates being combined in the analysis,^[15]^[7] as well as the specification of homogeneity as the null hypothesis which is then only rejected in the presence of sufficient evidence against it.^[16]

Estimation

While the main purpose of a meta-analysis usually is estimation of the main effect, investigation of the heterogeneity is also crucial for its interpretation. A large number of (frequentist and Bayesian) estimators is available.^[17] Bayesian estimation of the heterogeneity usually requires the specification of an appropriate prior distribution.^[9]^[18]

While many of these estimators behave similarly in case of a large number of studies, differences in particular arise in their behaviour in the common case of only few estimates.^[19] An incorrect zero between-study variance estimate is frequently obtained, leading to a false homogeneity assumption. Overall, it appears that heterogeneity is being consistently underestimated in meta-analyses.^[7]

Quantification

The heterogeneity variance is commonly denoted by τ², or the standard deviation (its square root) by τ. Heterogeneity is probably most readily interpretable in terms of τ, as this is the heterogeneity distribution's scale parameter, which is measured in the same units as the overall effect itself.^[18]

Another common measure of heterogeneity is I², a statistic that indicates the percentage of variance in a meta-analysis that is attributable to study heterogeneity (somewhat similarly to a coefficient of determination).^[20] I² relates the heterogeneity variance's magnitude to the size of the individual estimates' variances (squared standard errors); with this normalisation however, it is not quite obvious what exactly would constitute "small" or "large" amounts of heterogeneity. For a constant heterogeneity (τ), the availability of smaller or larger studies (with correspondingly differing standard errors associated) would affect the I² measure; so the actual interpretation of an I² value is not straightforward.^[21] ^[22]

The joint consideration of a prediction interval along with a confidence interval for the main effect may help getting a better sense of the contribution of heterogeneity to the uncertainty around the effect estimate.^[5]^[23]^[24]^[25]

References

^ ^a ^b Deeks, J.J.; Higgins, J.P.T.; Altman, D.G. (2021), "10.10 Heterogeneity", in Higgins, J.P.T.; Thomas, J.; Chandler, J.; Cumpston, M.; Li, T.; Page, M.J.; Welch, V.A. (eds.), Cochrane Handbook for Systematic Reviews of Interventions (6.2 ed.)
^ Singh, A.; Hussain, S.; Najmi, A.N. (2017), "Number of studies, heterogeneity, generalisability, and the choice of method for meta-analysis", Journal of the Neurological Sciences, 15 (381): 347, doi:10.1016/j.jns.2017.09.026, PMID 28967410, S2CID 31073171
^ Deeks, J.J.; Altman, D.G. (2001), "Effect measures for meta-analysis of trials with binary outcomes", in Egger, M.; Davey Smith, G.; Altman, D. (eds.), Systematic reviews in health care: Meta-analysis in context (2nd ed.), BMJ Publishing, pp. 313–335, doi:10.1002/9780470693926.ch16, ISBN 9780470693926
^ Cooper, Harris; Hedges, Larry V.; Valentine, Jeffrey C. (2019-06-14). The Handbook of Research Synthesis and Meta-Analysis. Russell Sage Foundation. ISBN 978-1-61044-886-4.
^ ^a ^b Riley, R. D.; Higgins, J. P.; Deeks, J. J. (2011), "Interpretation of random-effects meta-analyses", BMJ, 342: d549, doi:10.1136/bmj.d549, PMID 21310794, S2CID 32994689
^ Bretthorst, G.L. (1999), "The near-irrelevance of sampling frequency distributions", in von der Linden, W.; et al. (eds.), Maximum Entropy and Bayesian methods, Kluwer Academic Publishers, pp. 21–46, doi:10.1007/978-94-011-4710-1_3, ISBN 978-94-010-5982-4
^ ^a ^b ^c Kontopantelis, E.; Springate, D. A.; Reeves, D. (2013). "A re-analysis of the Cochrane Library data: The dangers of unobserved heterogeneity in meta-analyses". PLOS ONE. 8 (7): e69930. Bibcode:2013PLoSO...869930K. doi:10.1371/journal.pone.0069930. PMC 3724681. PMID 23922860.
^ Kontopantelis, E.; Reeves, D. (2012). "Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study". Statistical Methods in Medical Research. 21 (4): 409–26. doi:10.1177/0962280210392008. PMID 21148194. S2CID 152379.
^ ^a ^b Röver, C. (2020), "Bayesian random-effects meta-analysis using the bayesmeta R package", Journal of Statistical Software, 93 (6): 1–51, arXiv:1711.08683, doi:10.18637/jss.v093.i06
^ Cornell, John E.; Mulrow, Cynthia D.; Localio, Russell; Stack, Catharine B.; Meibohm, Anne R.; Guallar, Eliseo; Goodman, Steven N. (2014-02-18). "Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change". Annals of Internal Medicine. 160 (4): 267–270. doi:10.7326/M13-2886. ISSN 0003-4819. PMID 24727843. S2CID 9210956.
^ Maziarz, Mariusz (2022-02-01). "Is meta-analysis of RCTs assessing the efficacy of interventions a reliable source of evidence for therapeutic decisions?". Studies in History and Philosophy of Science. 91: 159–167. doi:10.1016/j.shpsa.2021.11.007. ISSN 0039-3681. PMID 34922183. S2CID 245241150.
^ Borenstein, Michael; Hedges, Larry V.; Higgins, Julian P. T.; Rothstein, Hannah R. (2010). "A basic introduction to fixed-effect and random-effects models for meta-analysis". Research Synthesis Methods. 1 (2): 97–111. doi:10.1002/jrsm.12. ISSN 1759-2887. PMID 26061376. S2CID 1040498.
^ Cochran, W.G. (1954), "The combination of estimates from different experiments", Biometrics, 10 (1): 101–129, doi:10.2307/3001666, JSTOR 3001666
^ Hardy, R.J.; Thompson, S.G. (1998), "Detecting and describing heterogeneity in meta-analysis", Statistics in Medicine, 17 (8): 841–856, doi:10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-D, PMID 9595615
^ Davey, J.; Turner, R.M.; Clarke, M.J.; Higgins, J.P.T. (2011), "Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis", BMC Medical Research Methodology, 11 (1): 160, doi:10.1186/1471-2288-11-160, PMC 3247075, PMID 22114982
^ Li, W.; Liu, F.; Snavely, D. (2020), "Revisit of test‐then‐pool methods and some practical considerations", Pharmaceutical Statistics, 19 (5): 498–517, doi:10.1002/pst.2009, PMID 32171048, S2CID 212718520
^ Veroniki, A.A.; Jackson, D.; Viechtbauer, W.; Bender, R.; Bowden, J.; Knapp, G.; Kuß, O.; Higgins, J.P.T.; Langan, D.; Salanti, G. (2016), "Methods to estimate the between-study variance and its uncertainty in meta-analysis", Research Synthesis Methods, 7 (1): 55–79, doi:10.1002/jrsm.1164, PMC 4950030, PMID 26332144
^ ^a ^b Röver, C.; Bender, R.; Dias, S.; Schmid, C.H.; Schmidli, H.; Sturtz, S.; Weber, S.; Friede, T. (2021), "On weakly informative prior distributions for the heterogeneity parameter in Bayesian random‐effects meta‐analysis", Research Synthesis Methods, 12 (4): 448–474, arXiv:2007.08352, doi:10.1002/jrsm.1475, PMID 33486828, S2CID 220546288
^ Friede, T.; Röver, C.; Wandel, S.; Neuenschwander, B. (2017), "Meta-analysis of few small studies in orphan diseases", Research Synthesis Methods, 8 (1): 79–91, arXiv:1601.06533, doi:10.1002/jrsm.1217, PMC 5347842, PMID 27362487
^ Higgins, J. P. T.; Thompson, S. G.; Deeks, J. J.; Altman, D. G. (2003), "Measuring inconsistency in meta-analyses", BMJ, 327 (7414): 557–560, doi:10.1136/bmj.327.7414.557, PMC 192859, PMID 12958120
^ Rücker, G.; Schwarzer, G.; Carpenter, J.R.; Schumacher, M. (2008), "Undue reliance on I² in assessing heterogeneity may mislead", BMC Medical Research Methodology, 8 (79): 79, doi:10.1186/1471-2288-8-79, PMC 2648991, PMID 19036172
^ Borenstein, M.; Higgins, J.P.T.; Hedges, L.V.; Rothstein, H.R. (2017), "Basics of meta-analysis: I² is not an absolute measure of heterogeneity" (PDF), Research Synthesis Methods, 8 (1): 5–18, doi:10.1002/jrsm.1230, hdl:1983/9cea2307-8e9b-4583-9403-3a37409ed1cb, PMID 28058794, S2CID 4235538
^ Chiolero, A; Santschi, V.; Burnand, B.; Platt, R.W.; Paradis, G. (2012), "Meta-analyses: with confidence or prediction intervals?" (PDF), European Journal of Epidemiology, 27 (10): 823–5, doi:10.1007/s10654-012-9738-y, PMID 23070657, S2CID 20413290
^ Bender, R.; Kuß, O.; Koch, A.; Schwenke, C.; Hauschke, D. (2014), Application of prediction intervals in meta-analyses with random effects (PDF), Joint statement of IQWiG, GMDS and IBS-DR
^ IntHout, J; Ioannidis, J.P.A.; Rovers, M.M.; Goeman, J.J. (2016), "Plea for routinely presenting prediction intervals in meta-analysis" (PDF), BMJ Open, 6 (7): e010247, doi:10.1136/bmjopen-2015-010247, PMC 4947751, PMID 27406637