Advertisement intended for healthcare professionals

European Renal Association - European Dialysis and Transplant Association

Navbar Search Filter Nephrology Dialysis TransplantationThis issueERA Journals NephrologyBooksJournalsOxford Academic Enter search term

AI Discovery Assistant

Volume 31

Issue 6

June 2016

Article Contents

Comments (0)

Journal Article

Con: Meta-analysis: some key limitations and potential solutions

Tonya M. Esterhuizen, Lehana Thabane

Nephrology Dialysis Transplantation, Volume 31, Issue 6, June 2016, Pages 882–885, https://doi.org/10.1093/ndt/gfw092

Published:

20 May 2016

Article history

Abstract

Meta-analysis, a statistical combination of results of several trials to produce a summary effect, has been subject to criticism in the past, mainly for the reasons of poor quality of included studies, heterogeneity between studies meta-analyzed and failing to address publication bias. These limitations can cause the results to be misleading, which is important if policy and practice decisions are based on systematic reviews and meta-analyses. We elaborate on these limitations and illustrate them with examples from the nephrology literature. Finally, we present some potential solutions, notably, education in meta-analysis for evidence producers and consumers as well as the use of individual patient data for meta-analyses.

evidence-based medicine, heterogeneity, limitations, meta-analysis, publication bias

Topic:

Issue Section:

Cutting-Edge Renal Science > POLAR VIEWS IN NEPHROLOGY

INTRODUCTION

While there is general agreement in the evidence-based decision-making paradigm of the place that systematic reviews hold at the tip of the pyramid of the hierarchy of evidence, the role and usefulness of meta-analysis need to be clarified. Systematic reviews, when conducted using rigorous and clear methods, with attention to detail on the quality of included studies, can provide very useful overviews on the current level of knowledge on a particular topic. Meta-analyses, when conduced from within the context of these well-conducted systematic reviews, and if done appropriately, can provide summary estimates of the effect of interventions from several similar studies.

But meta-analysis has had its fair share of criticism over the years, some calling it ‘mega silliness’ [1], ‘statistical alchemy’ [2] and ‘meta-analysis/schmeta analysis’ [3]. In a BMJ article, Eysenck wrote: ‘If a treatment has an effect so recondite and obscure as to require meta-analysis to establish it, I would not be happy to have it used on me’ [4].

The use of meta-analysis in nephrology has increased in parallel with an increase in clinical trials in this field over the years [5, 6]. A PubMed search using the MeSH terms ‘meta-analysis’ as a publication type or topic and kidney disease, filtered by year from 2004 to 2013, revealed a 6-fold increase in the number of hits over the years, as shown in Figure 1. In a 2010 survey of nephrologists, meta-analyses were starting to be used as a source of evidence-based care and to influence individual patient care [5]. If practitioners are using meta-analyses as a basis for their management decisions, it is important that the limitations are well understood.

Hits on a PubMed search for meta-analyses in nephrology between 2004 and 2013.

FIGURE 1:

Hits on a PubMed search for meta-analyses in nephrology between 2004 and 2013.

Open in new tab Download slide

The objective of this article is to briefly describe some of the limitations of meta-analysis and ways to address them. We focus on poor quality of included studies, heterogeneity and publication bias, because these criticisms have been found to be frequently referred to in reviews and overviews describing the strengths and limitations of meta-analyses [6–8]; however, this is not an exhaustive list of the criticisms. Where available, examples from meta-analyses in nephrology will be used to illustrate these limitations. Note that when we refer to meta-analysis here, unless otherwise stated, we are referring to aggregated patient data (APD) meta-analyses.

EXAMPLES OF LIMITATIONS OR CRITICISMS

Poor quality of included studies

A well-done meta-analysis of badly designed studies will yield invalid results. Bias and confounding in the primary studies are major problems for meta-analysis, notably, issues of randomization and blinding in clinical trials. If the statistical analysis used in the primary study was faulty, including the summary statistics from that study in an APD meta-analysis will lead to flawed results. Additionally, meta-analyses can be poorly executed. Carelessness in abstracting and summarizing appropriate studies, failure to consider important covariates, bias on the part of the meta-analyst and overstatements of the strength and precision of the results can all contribute to invalid meta-analyses.

In a systematic review and meta-analysis of the effects if vitamin D therapy on biochemical markers of cardiovascular outcomes in CKD patients, the authors found that allocation concealment was not even reported in 67% of the 76 eligible randomized controlled trials, and blinding was reported in only 34% [9]. Another study that looked at methodological quality of nephrology-related systematic reviews, published in 2005 [10], found that 54% of the 90 reviews suffered from major methodological flaws, notably failing to assess the methodological quality of included studies (49%).

The Consolidated Standards of Reporting Trials (CONSORT) statement, introduced in 2001 [11], specifies the required information that should be reported in clinical trials. However, even today, not all trials are reported to these standards, making it difficult for meta-analysts to properly assess their quality [12]. To avoid the problem of including low-quality studies, the meta-analyst should explore differences in study quality. This can be done using sensitivity analysis, by assessing the effect of excluding studies with certain methodological weaknesses on the summary effect in the meta-analysis.

Education of producers and consumers of meta-analysis is essential, as well as the realization that the summary statistic produced is not a magic number. Critical appraisal of meta-analyses should be encouraged rather than accepting them at face value. Meta-analysis is a statistically and methodologically complex process that often the users of the evidence do not fully understand. Mrkobra et al. [10] found that reviews published in journals that endorsed consensus guidelines such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines [13] had higher quality with fewer methodological flaws. In a review article, Yuan and Hunt [8] point out the ‘ugly’ consequences of meta-analysis that could arise from the three limitations mentioned above, among others. These include meta-analysis being deceptive due to errors made in the analysis and heterogeneity of studies and misuse of meta-analysis due to results being extrapolated to populations not originally included in the meta-analysis [8]. They conclude that in order to make informed clinical decisions, clinicians should take into account the strength of the evidence, methodology and validity of the meta-analysis and systematic review.

Heterogeneity

‘Combining apples and oranges’ has been commonly used as a metaphor to describe the problem of pooling studies that are dissimilar in some ways. The potential to ignore important differences between studies can invalidate meta-analysis. Some feel that meta-analysis should not even be the ultimate goal of a systematic review. Pooling may not be appropriate if effects are not robust or consistent across studies.

Possible clinical heterogeneity was reported in a review of a meta-analysis of the effect of statins in CKD patients at various stages of disease on cardiovascular endpoints [14]. The authors pooled studies using different types of statins. The results of this meta-analysis were judged to have little relevance to clinicians due to important differences in treatment effects across different subgroups being obscured [14].

Statistical heterogeneity was clearly evident in a meta-analysis of the use of acetylcysteine versus placebo for prevention of contrast nephropathy [15] for the outcome of serum creatinine after 48 h. The forest plot showed non-overlapping confidence intervals for the seven included studies and a suggestion that the smaller studies showed more benefit. However, no heterogeneity statistic was reported and a summary mean difference was calculated and reported. When we conducted a random effects meta-analysis on these data, we found that the I² statistic was in fact 91%, χ² P < 0.00001, indicating the presence of statistical heterogeneity, or that 91% of the observed variance reflects differences in effect size between studies (Figure 2). The usefulness of the summary statistic is questionable in this situation, where the source of the heterogeneity should have been investigated prior to meta-analysis.

Reanalysis of the weighted mean difference of serum creatinine (mg/dl) at 48 h: N-acetylcysteine versus placebo. Data from Isenbarger et al. [15].

FIGURE 2:

Reanalysis of the weighted mean difference of serum creatinine (mg/dl) at 48 h: N-acetylcysteine versus placebo. Data from Isenbarger et al. [15].

Open in new tab Download slide

If modest heterogeneity exists (I² statistic between 30 and 60% [16]), this should stimulate thought about the origin of the variation, whether it is clinical or statistical. If heterogeneity is substantial, the focus should be on exploring and understanding the sources of variation, and pooling of the data in a meta-analysis may not be appropriate. Subgroup analyses and meta-regression are tools available for the purpose of exploring the sources of the heterogeneity and should be conducted in collaboration with a statistician.

Individual patient data (IPD) meta-analysis can potentially overcome some of the problems of APD meta-analysis. It avoids biases associated with combining the summary statistics of separate studies, enables adjustment for individual level confounders and circumvents dependence on the already-performed statistical analysis [17]. Moreover, IPD meta-analyses usually require a lot more resources to conduct than ADP meta-analyses, and access to IPD may be a challenge. A comparison of the strengths and limitations of APD meta-analysis versus IPD meta-analysis concluded that while IPD meta-analysis offers many statistical advantages over APD meta-analysis, the time and cost saving of properly performed APD meta-analysis might outweigh these benefits [7].

Publication bias

Also known as the ‘file drawer problem’ [18], this well-known phenomenon is where studies that showed no effect or were not statistically significant are not likely to be published and therefore do not appear in meta-analyses, which sometimes only search the published literature. Publication bias is likely to lead to overestimation of the true effect in meta-analysis.

Another meta-analysis on the effect of acetylcysteine for prevention of contrast nephropathy [19] revealed a suspected publication bias in the absence of smaller studies from the funnel plot and confirmed this with a statistical test. The authors hypothesized that the absence of small trials with negative results could have skewed their findings toward a positive result, despite searching the published and unpublished literature [19].

Avoidance of publication bias in meta-analysis is the key to prevention. As far as possible, unpublished material should be sourced for inclusion. However, identification of unpublished results is often problematic. Trial registries may go a long way towards overcoming this problem by allowing reviewers to identify all trials conducted on particular interventions and thus flag trials where no results were published. If publication bias is suspected, it can sometimes be detected through funnel plots and corresponding statistics. Funnel plots are scatter plots of the individual studies’ effect size against precision. Symmetrical plots tend to indicate no real publication bias, but where the plot is asymmetrical, it can sometimes indicate reporting or other biases [16]. Where it is present, the effect of publication bias can be seen by conducting sensitivity analysis, the comparison of different meta-analysis models under different assumptions. More information can be found in the Cochrane handbook [16]. However, there is nothing much that can be done to remedy the effects of publication bias, not even the use of IPD.

Finally, the Cochrane Collaboration [20] provides the platform and support, including training, for systematic reviews and meta-analysis production, dissemination and use, and the Cochrane Library is available free of charge in developing countries. Rigorous peer review of protocols and reviews takes place within this collaboration. Systematic reviews and meta-analysis should be conducted within this framework where possible.

CONFLICT OF INTEREST STATEMENT

We declare that the contents of this paper have not been published previously in whole or part, except in abstract format.

REFERENCES

Eysenck

An exercise in mega-silliness

Am Psychol

1978

;

517

Article Contents

Con: Meta-analysis: some key limitations and potential solutions

Abstract

INTRODUCTION

EXAMPLES OF LIMITATIONS OR CRITICISMS

Poor quality of included studies

Heterogeneity

Publication bias

CONFLICT OF INTEREST STATEMENT

REFERENCES

Comments

Citations

Views

Altmetric

Email alerts

Citing articles via

Share this: