For practical reasons, we present a stepwise description of the tasks that are performed when statistical methods are used to combine data. These tasks are 1) deciding whether to combine data and defining what to combine, 2) evaluating the statistical heterogeneity of the data, 3) estimating a common effect, 4) exploring and explaining heterogeneity, 5) assessing the potential for bias, and 6) presenting the results.

Deciding Whether To Combine Data and Defining What To Combine

By the time one performs a quantitative synthesis, certain decisions should already have been made about the formulation of the question and the selection of included studies. These topics were discussed in two previous articles in this series [1, 2]. Statistical tests cannot compensate for lack of common sense, clinical acumen, and biological plausibility in the design of the protocol of a meta-analysis. Thus, a reader of a systematic review should always address these issues before evaluating the statistical methods that have been used and the results that have been generated. Combining poor-quality data, overly biased data, or data that do not make sense can easily produce unreliable results.

The data to be combined in a meta-analysis are usually either binary or continuous. Binary data involve a yes/no categorization (for example, death or survival). Continuous data take a range of values (for example, change in diastolic blood pressure after antihypertensive treatment, measured in mm Hg).

When one is comparing groups of patients, binary data can be summarized by using several measures of treatment effect that were discussed earlier in this series [3]. These measures include the risk ratio; the odds ratio; the risk difference; and, when study duration is important, the incidence rate. Another useful clinical measure, the number needed to treat (NNT), is derived from the inverse of the risk difference [3]. Treatment effect measures, such as the risk ratio and the odds ratio, provide an estimate of the relative efficacy of an intervention, whereas the risk difference describes the intervention's absolute benefit. The various measures of treatment effect offer complementary information, and all should be examined [4].

Continuous data can be summarized by the raw mean difference between the treatment and control groups when the treatment effect is measured on the same scale (for example, diastolic blood pressure in mm Hg), by the standardized mean difference when different scales are used to measure the same treatment effect (for example, different pain scales being combined), or by the correlation coefficients between two continuous variables [5]. The standardized mean difference, also called the effect size, is obtained by dividing the difference between the mean in the treatment group and the mean in the control group by the SD in the control group.

Evaluating the Statistical Heterogeneity of the Data

This step is intended to answer the question, Are the results of the different studies similar (homogeneous)? It is important to answer this question before combining any data. To do this, one must calculate the magnitude of the statistical diversity (heterogeneity) of the treatment effect that exists among the different sets of data.

Statistical diversity can be thought of as attributable to one or both of two causes. First, study results can differ because of random sampling error. Even if the true effect is the same in each study, the results of different studies would be expected to vary randomly around the true common fixed effect. This diversity is called the within-study variance. Second, each study may have been drawn from a different population, depending on the particular patients chosen and the interventions and conditions unique to the study. Therefore, even if each study enrolled a large patient sample, the treatment effect would be expected to differ. These differences, called random effects, describe the between-study variation with regard to an overall mean of the effects of all of the studies that could be undertaken.

The test most commonly used to assess the statistical significance of between-study heterogeneity is based on the chi-square distribution [6]. It provides a measure of the sum of the squared differences between the results observed and the results expected in each study, under the assumption that each study estimates the same common treatment effect. A large total deviation indicates that a single common treatment effect is unlikely. Any pooled estimate calculated must account for the between-study heterogeneity. In practice, this test has low sensitivity for detecting heterogeneity, and it has been suggested that a liberal significance level, such as 0.1, should be used [6].

Estimating a Common Effect

The questions that this step tries to answers are, 1) To the extent that data are similar, what is their best common point estimate of a therapeutic effect, and 2) how precise is this estimate? The mathematical process involved in this step generally involves combining (pooling) the results of different studies into an overall estimate. Compared with the results of individual studies, pooled results can increase statistical power and lead to more precise estimates of treatment effect.

Each study is given a weight according to the precision of its results. The rationale is that studies with narrow CIs should be weighted more heavily than studies with greater uncertainty. The precision is generally expressed by the inverse of the variance of the estimate of each study. The variance has two components: the variance of the individual study and the variance between different studies. When the between-study variance is found to be or assumed to be zero, each study is simply weighted by the inverse of its own variance, which is a function of the study size and the number of events in the study. This approach characterizes a fixed-effects model, as exemplified by the Mantel-Haenszel method [7, 8] or the Peto method [9] for dichotomous data. The Peto method has been particularly popular in the past. It has the advantage of simple calculation; however, although it is appropriate in most cases, it may introduce large biases if the data are unbalanced [10, 11]. On the other hand, random-effects models also add the between-study variance to the within-study variance of each individual study when the pooled mean of the random effects is calculated. The random-effects model most commonly used for dichotomous data is the DerSimonian and Laird estimate of the between-study variance [12]. Fixed- and random-effects models for continuous data have also been described [13]. Pooled results are generally reported as a point estimate and CI, typically a 95% CI.

Other quantitative techniques for combining data, such as the Confidence Profile Method [14], use Bayesian methods to calculate posterior probability distributions for effects of interest. Bayesian statistics are based on the principle that each observation or set of observations should be viewed in conjunction with a prior probability describing the prior knowledge about the phenomenon of interest [15]. The new observations alter this prior probability to generate a posterior probability. Traditional meta-analysis assumes that nothing is known about the magnitude of the treatment effect before randomized trials are performed. In Bayesian terms, the prior probability distribution is noninformative. Bayesian approaches may also allow the incorporation of indirect evidence in generating prior distributions [14] and may be particularly helpful in situations in which few data from randomized studies exist [16]. Bayesian analyses may also be used to account for the uncertainty introduced by estimating the between-study variance in the random-effects model, leading to more appropriate estimates and predictions of treatment efficacy [17].

Exploring and Explaining Heterogeneity

The next important issue is whether the common estimate obtained in the previous step is robust. Sensitivity analyses determine whether the common estimate is influenced by changes in the assumptions and in the protocol for combining the data.

A comparison of the results of fixed- and random-effects models is one such sensitivity analysis [18]. Generally, the random-effects model produces wider CIs than does the fixed-effects model, and the level of statistical significance may therefore be different depending on the model used. The pooled point estimate per se is less likely to be affected, although exceptions are possible [19].

Other sensitivity analyses may include the examination of the residuals and the chi-square components [13] and assessment of the effect of deleting each study in turn. Statistically significant results that depend on a single study may require further exploration.

Cumulative Meta-Analysis

Cumulative meta-analysis is another approach for assessing the impact of each study [20]; it is the opposite of the stepwise deletion. In cumulative meta-analysis, studies are sequentially pooled by adding one study at a time in a prespecified order [21]. One possible order is according to the dates these studies were conducted or published. Cumulative meta-analysis can help determine whether the pooled estimate has been robust over time and can also determine the point in time when statistical significance was reached for a pooled result. When the order is the year of publication, cumulative meta-analysis can be seen as a form of Bayesian inference. The prior probability (the prior belief) is generated by the pooled results of all prior studies, and the posterior probability is derived by adding the results of the new study to the results of the others [21].

Meta-Regression

Further sensitivity analyses are generally dictated by the nature and the specifics of the question that the meta-analysis tries to answer and by the possible reasons that can be identified to explain heterogeneity. One such computational procedure, commonly referred to as meta-regression, involves the statistical assessment of whether specific factors (covariates) influence the magnitude of the point estimate of the treatment effect across studies [22]. Meta-regression results are generally reported as slope coefficients with CIs. The covariates of interest may describe study or patient characteristics. These characteristics may be common for all patients in each study (for example, the specific route of administration of the experimental drug used in each study) or they may be average values representative of the studied cohort (such as the mean age of the patients). Averages of covariates measured at the patient level require cautious interpretation because the aggregate values may not adequately represent important minorities of patients [23-25].

Some covariates are ubiquitous, such as study sample size, study result variance, and control rate of events (the percentage of patients with an event of interest in the control group). Other covariates may be problem-specific. Often, information on covariates may not be uniformly collected or reported across all studies, and analyses involving these covariates may therefore not be useful. A variety of statistical methods, including weighted least-squares, logistic regression, and hierarchical models, can be used for meta-regression analyses [22, 26-28].

Figure 1 summarizes the three previous steps, which define the core of a meta-analysis. It shows that estimating, deciding whether to ignore, incorporating, exploring, and explaining heterogeneity are the key aims of the quantitative methods for synthesizing data from different studies.

View larger version (20K): [in this window] [in a new window]   Figure 1. Methodologic choices and their implications in dealing with heterogeneous data in a meta-analysis.

Subgroup Analysis

Subgroup analyses may be useful for addressing particular questions when data for different subgroups of patients are available from each study [29]. Combining specific subgroup data across studies follows the principles described above and may provide further insight into heterogeneity. Subgroup analyses in the retrospective setting of most meta-analyses are post hoc exercises and should be interpreted with caution, lest they turn into "fishing expeditions." An especially pernicious approach occurs when the data are divided into multiple subgroups on the basis of combinations of characteristics (such as age and dose) and differential treatment effects are claimed within very small subdivisions. Such interactions among subgroups are unlikely to describe the truth when derived from aggregated data.

Lack of uniform reporting of the data necessary for subgroup analyses across trials poses an additional problem. Thus, subgroup analyses should best be used as hypothesis-generating tools [22], although important observations may sometimes be made [30].

Assessing the Potential for Bias

The assessment of potential bias should always be part of a meta-analysis. Some issues relevant to this were discussed earlier in this series [1, 2]. Two major sources of bias for meta-analysis are the failure to find all of the studies performed in the clinical domain and the uncertain reliability of poor-quality studies.

Publication Bias

Studies with negative results are more likely to remain unpublished because investigators or the peer reviewers and editors are not enthusiastic about publishing "negative" information [31-33]. The chances of not being published are probably greater if the negative study is small and nonrandomized [34]. Some studies may be impossible to retrieve and include in a meta-analysis despite a thorough search of potential databases. Publication bias is difficult to eliminate, but some statistical procedures may be helpful in detecting its presence. An inverted funnel plot [35] is sometimes used to visually explore the possibility that publication bias is present (Figure 2). This method uses a scatterplot of studies that relates the magnitude of the treatment effect to the weight of the study. An inverted, funnel-shaped, symmetrical appearance of dots suggests that no study has been left out, whereas an asymmetrical appearance suggests the presence of publication bias. Formal computational approaches to test for, assess the extent of, and correct publication bias have also been described [36-39].

View larger version (17K): [in this window] [in a new window]   Figure 2. An inverted funnel plot to detect publication bias. This example used data from a meta-analysis of intravenous streptokinase for acute myocardial infarction [20]. The risk ratio for the mortality reduction in each study is plotted against the weight of the study, represented by the sample size. A symmetric triangle is fitted around the pooled estimate (arrow) so that it encompasses most of the studies. If small "negative" trials with large variance have been left unpublished, the plot will be asymmetrical: Small published studies will show very large estimates of the treatment effect compared with larger studies that have more conservative results. A symmetrical plot provides reassuring evidence that the treatment effect is similar in studies of small and large variance, whereas an asymmetrical plot suggests possible publication bias. The plot shown here reveals that there are fewer small studies (involving 10 to 100 participants) with risk ratios greater than 0.8 than there are small studies with risk ratios less than 0.8, whereas the numbers of medium and large studies are fairly symmetrical. These results suggest that some small studies with negative findings were not published. Outlier studies may also be readily identified by using this plot.

Quality

Study quality was discussed in detail earlier in this series [2]. Investigators have proposed incorporating quality scores into meta-analyses on the basis of checklists of study design components [40-43]. To date, no scale has been proven to correlate consistently with treatment efficacy [44]. Beyond the generic features of study design and conduct, general quality-scoring systems may have to be supplemented or replaced with more problem-specific quality items for each particular meta-analysis [45]. Empirical investigations have shown that studies of worse quality may overestimate treatment effects because they inadequately conceal treatment allocation and use inadequate blinding [46].

Presenting the Results

The results of meta-analyses are typically presented in a graphic form (Figure 3) that shows the point estimates and their CIs. This presentation aims to convey an impression of the results of the individual studies, to convey the extent of heterogeneity, and to report the pooled estimate. Meta-regression analyses may be depicted by plots in which the value of the covariate of interest is shown on the horizontal axis and the magnitude of the treatment effect is shown on the vertical axis [48]. It is also important to report results from sensitivity analyses on key issues and comparison between fixed-effects and random-effects methods of pooling data when their results are different. When appropriate, reporting the NNT helps translate the results into a more clinically meaningful metric [3].

View larger version (26K): [in this window] [in a new window]   Figure 3. Standard meta-analysis and cumulative meta-analysis, Left. A standard meta-analysis plot of the risk ratios for progression to AIDS or death in a comparison of early therapy with zidovudine (treatment group) or deferred therapy with zidovudine (control group) [47]. The point estimates for the risk ratio of each study and the pooled point estimate are shown by the points, and the horizontal lines show the CIs, typically 95% CIs. N is the number of patients in the study. The studies are ordered according to year of publication. As a standard convention, a risk ratio of less than 1 denotes a reduction in the number of events in the treated compared with the control group. Right. The results of a cumulative meta-analysis of the same data. N is the number of patients in the clinical trials. The points and lines represent the point estimates and the 95% CIs of the pooled results after the inclusion of each additional study in the calculations. The CIs typically narrow with the addition of more studies unless substantial heterogeneity exists.

Other Types of Data and Methods

Meta-Analysis of Diagnostic Tests

An important application of meta-analysis is the combination of sensitivity and specificity data of diagnostic tests across different studies [49]. Using weighted linear regression to generate a summary receiver-operating characteristics (ROC) curve has been proposed as a way to avoid the underestimation of test performance that results when the correlation between sensitivity and specificity is ignored [50]. An ROC curve is a plot of the percentage of true-positive results (the sensitivity of the test) against the percentage of false-positive results (1 –specificity) and thus represents the tradeoff between these two test characteristics.

Meta-Analysis of Other Nonrandomized, Uncontrolled Data

Uncontrolled cohort data can also be combined by using meta-analytic techniques. The principles are the same as those described for randomized data. However, greater care is needed in the conduct of the analysis and interpretation of the results when nonrandomized and uncontrolled data are used because these data are more likely to be biased. Of particular interest is the synthesis of dose-response data across different studies that investigate the effect of increasing values of a potential etiologic factor on an outcome of interest (for example, exposure to environmental tobacco smoke and the occurrence of lung cancer) [17, 51-53].

Meta-Analysis of Individual Patient Data

Most meta-analyses are based on group data as reported in the literature, but researchers occasionally make the effort to collect the detailed outcomes and risk factor data for the individual patients involved in each of several studies. These data can be used in survival analyses and multivariate regression analyses. Meta-analysis of individual patient data is more expensive and time-consuming than meta-analysis of grouped data, and it requires the coordination of large teams of investigators and a robust protocol [54]. Nevertheless, if possible, meta-analysis of individual patient data may represent the highest step in the hierarchy of evidence [55].

Conclusions

By quantitatively summarizing a collection of data from clinical studies, meta-analysis provides collective results of a kind that no individual study can offer. Meta-analysis is a relatively new discipline in clinical medicine. As might be expected, discrepancies between the results of large trials and meta-analyses of smaller trials [19], as well as differences in the results of meta-analyses addressing the same topic, do occur [56, 57]. New quantitative methods are being developed, and investigators have addressed these important issues [58-60]. Synthesis of the data from many individual studies requires sound, rigorous, quantitative methods, and the results of such syntheses should be interpreted with appropriate caution; meta-analysis is not a "magic" solution to the problem of scientific evidence and cannot replace clinical reasoning [61]. In addition, reliable meta-analysis requires consistent, high-quality reporting of the primary data from individual studies; the need for such reporting cannot be overstated [62, 63]. (Table 1)

Glossary

Top Glossary Author & Article Info References

Bayesian inference: A statistical discipline that addresses how a prior estimate should be modified in the light of knowledge gained from new studies.

Cumulative meta-analysis: A method whereby the combined point estimate of an effect is sequentially computed by adding one study at a time in a prespecified order.

Fixed-effects model: A model that assumes that all studies are studying the same true effect and that variability is due to random error only.

Heterogeneity: The diversity that exists between studies. It may be due to identifiable factors or statistical factors, or both, especially the component that cannot be explained by random error.

Meta-regression: A regression analysis in which individual sets of data (studies) are used as the unit of observation.

Random-effects model: A model that assumes that the true effect differs among studies and therefore must be represented by a distribution of values instead of a single value.

Receiver-operating characteristic curve: A plot of the characteristics of a diagnostic test. It depicts the tradeoff between the sensitivity and the specificity of the test.

Dr. Ioannidis: Therapeutics Research Program, Division of AIDS, National Institute of Allergy and Infectious Diseases, Solar Building, Room 2C15, Bethesda, MD 20892.