Meta-analysis, however, is a statistical tool with inherent strengths and weaknesses (Table 1). Clinically, it is most commonly and most appropriately used to pool data from available small studies in order to generate hypotheses and calculate the sample size estimates needed for definitive trials. Because meta-analyses rely on previously published data, they are inherently observational rather than experimental. Pooled results do not account for epidemiologic biases present in the original studies, and they may even introduce new bias by inappropriately pooling heterogeneous studies. Additionally, various techniques for meta-analysis are available, and these have the potential to result in apparently conflicting summary estimates, even when they are used to combine results from the same studies. Thus, the clinical interpretation of meta-analyses must be done with appropriate caution. Physicians treating patients need to know that a meta-analysis of 10 studies of 100 patients each is not a substitute for data derived from a single randomized trial of 1000 patients [6].

Nonetheless, a major strength of meta-analysis is its ability to discern small but potentially important clinical effects. The technique has understandably found ready acceptance in the medical literature, because it can provide reliable treatment recommendations in the absence of data from definitive trials [3].

Within the field of internal medicine, meta-analysis has often been used in making decisions about the treatment of acute myocardial infarction [7-11]. For example, meta-analytic evidence supported the use of two widely adopted treatments for myocardial infarction that were subsequently confirmed by the results of large-scale randomized trials: early ß-blockers and thrombolytic therapy [5, 8, 9] (Figure 1). The meta-analytic evidence is also compelling for two other treatments: nitrates [10] and intravenous magnesium therapy [11]. However, enthusiasm for these two treatments has been considerably dampened [16] by the results from two large-scale trials, the Fourth International Study of Infarct Survival (ISIS-4) and the Third Gruppo Italiano per lo Studio della Sopravvivenza nell'Infarto Miocardico study (GISSI-3) [14, 15]. Specifically, the ISIS-4 investigators found that magnesium conferred no survival benefit in 58 000 patients with myocardial infarction who were randomly assigned to receive either high-dose intravenous magnesium or open control [15]. Further, these investigators found that no benefit was associated with oral nitrates compared with placebo, and the GISSI-3 investigators found no evidence to show that patients treated with intravenous nitroglycerin followed by topical nitrates experienced benefit compared with controls [13]. Together, these trials involved more than 70000 patients, far more than the 2000 patients accumulated in previous meta-analyses.

View larger version (19K): [in this window] [in a new window]   Figure 1. Concordance and discordance between meta-analyses and megatrials. Comparison of the odds ratios and 95% Cls for the given treatments. The first published meta-analysis is shown by the solid bars; the first published megatrial is shown by the open bars. Years of publication are given on the left; the numbers of patients included in each trial or meta-analysis are given on the right. Data on thrombolytic therapy is from references 8 and 12; data on early ß-blocker therapy is from references 9 and 13; data on nitrate therapy is from references 10 and 14; and data on magnesium therapy is from references 11 and 15.

Our purpose is to examine possible reasons for the discrepancy between meta-analytic and megatrial evidence about nitrate therapy and magnesium therapy. Although we focus on treatment decisions related to the patient with acute myocardial infarction, the issues we raise are applicable to various clinical settings in which there could be, or in which there is evidence of [17], discord between overview analyses and data from definitive trials.

Nitrate Therapy in Acute Myocardial Infarction

The use of nitrates to treat acute myocardial infarction is an example of the difficulties clinicians can face when interpreting meta-analytic evidence. On the basis of trials published between 1979 and 1985, Yusuf and colleagues [10] reviewed seven trials of intravenous nitroglycerin, which included between 28 and 310 patients, and three trials of intravenous nitroprusside, which included between 50 and 812 patients. Their meta-analysis of 2041 patients showed a survival odds ratio of 0.65 (95% CI, 0.51 to 0.82), favoring treatment with nitrates [10]. On the basis of their analysis, the authors appropriately suggested that routine nitrate use might be beneficial, and they called for a large-scale clinical trial to formally test this hypothesis.

Recently, two such trials—ISIS-4 [15] and GISSI-3 [14]—directly addressed this question. In the ISIS-4 trial, 58 050 patients with myocardial infarction who presented within 24 hours of symptom onset were randomly assigned to receive either placebo or oral isosorbide mononitrate, 30 mg initially and then 60 mg/d for 30 days. The principal study end point was 35-day mortality. There were 2190 deaths among the 29 032 placebo recipients (7.54%) and 2129 deaths among the 29 018 nitrate-treated patients (7.34%), producing a nonsignificant absolute mortality reduction (±SD) of 2.1 ± 2.2 per 1000 patients [15].

Similarly, the GISSI-3 investigators gave patients intravenous nitroglycerin followed by transdermal nitroglycerin patches for 6 weeks, to measure 6-week mortality. There were 653 deaths among 9442 controls (6.9%) and 617 deaths among 9453 nitrate-treated patients (6.5%); the difference again was nonsignificant [14]. When the two trials were combined, the mortality rates were 7.14% for controls and 6.87% for nitrate-treated patients, representing a nonsignificant reduction of 2.8 ± 1.9 per 1000 patients [15].

Several plausible explanations help to reconcile the discordant results of GISSI-3 and ISIS-4 and the previously available meta-analysis. First, both large-scale trials had substantial nontrial use of nitrates: 54% of patients in ISIS-4 and 57% of patients in GISSI-3. If nontrial nitrate use was random, such misclassification of exposure would have tended to decrease the apparent benefit of nitrates if a true benefit existed [18]. If nontrial nitrate use was nonrandom, it could have biased study outcome unpredictably.

Second, the trials composing the meta-analysis accounted for a total of fewer than 2000 patients and had wide Cls, whereas the large-scale, randomized clinical trials involved almost 80 000 patients. Thus, evidence from GISSI-3 and ISIS-4 provides much more stable estimates of effect than do the earlier studies.

Third, the difference between results may relate to changes in care of patients with myocardial infarction. The results of the early nitrate trials were published between 1979 and 1985, and the trials were necessarily completed 1 or 2 years before publication. An almost 50% decrease in in-hospital mortality from acute myocardial infarction occurred between 1980 and 1990 [19]; the widespread adoption of aspirin, ß-blocker therapy, thrombolytic therapy, and revascularization during that decade may at least partly explain this improvement [19]. Thus, the earlier trials examined nitrates in isolation, but ISIS-4 and GISSI-3 actually compared nitrate with no nitrate therapy in the setting of numerous other treatments: more than four fifths of patients in both trials received aspirin; one third (GISSI-3) to two thirds (ISIS-4) received thrombolytic agents; and one tenth (ISIS-4) to one third (GISSI-3) received intravenous ß-blockers [14, 15]. Finally, the earlier trials primarily used intravenous nitrates, whereas ISIS-4 and GISSI-3 (after the first 24 hours) used oral formulations, which may have been less effective.

Improving mortality rates and rapidly evolving treatment strategies may thus have led to difficulty in the selection of trials for inclusion in meta-analyses. Although the assumption is generally that all available data—published and unpublished as well as positive and negative—should be included [7], substantial heterogeneity may exist among studies on the basis of both study design and evolving practice. The tension between the principle that all available data should be included and the principle that a treatment should be examined in a contemporary, relatively homogeneous context is an inherent dilemma not easily resolved.

The different efficacy estimates from early meta-analyses and later trials of nitrate therapy also underscore a further distinction between overview analyses and large-scale randomized trials. When appropriately designed with adequate sample sizes to detect small but important treatment effects, randomized trials are inherently hypothesis-testing experiments, and their results should stand on their own as evidence for or against a particular intervention or therapy. In contrast, because meta-analyses rely on previously published data, they are inherently observational and should more properly be viewed as hypothesis-generating. Thus, meta-analyses may be considered "analytic" when designed to describe effect size, improve precision, or increase power and "exploratory" when designed to explain variation among effect sizes in previously published reports [20]. In neither case, however, can meta-analysis be considered experimental.

Magnesium Therapy in Acute Myocardial Infarction

The question of whether or not to use magnesium in the treatment of patients with myocardial infarction provides a second example of discordance between meta-analytic evidence and data derived from large-scale clinical trials. In 1991, Teo and coworkers [11] reviewed the seven trials of intravenous magnesium, which included between 48 and 400 patients and were reported between 1981 and 1990. The meta-analysis of 1301 patients showed an odds ratio of 0.45, favoring magnesium therapy, and the Cls ranged from one third to two thirds [11]. These results were reinforced by the Second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2), which showed a smaller mortality reduction of 24%, favoring magnesium therapy in 2316 patients [21], although the 95% Cls in this trial extended from a halving of mortality (compatible with the results of the meta-analysis) to almost no benefit. Unlike the smaller trials, in which adjunct treatment was not documented and was probably not given, one third of the patients in LIMIT-2 received thrombolytic agents, and two thirds received aspirin. When the LIMIT-2 trial was added to the meta-analysis, magnesium showed a 35% mortality reduction over control [21]. This led to the endorsement of magnesium therapy, by subsequent meta-analyses and persuasive editorials, for routine use [22].

To formally test in a very large population the hypothesis that magnesium therapy is effective, the ISIS-4 investigators randomly assigned more than 58 000 patients to receive either intravenous magnesium in a dose similar to that used in LIMIT-2 or open control. There were 2103 deaths among 29 039 patients assigned to receive open control (7.24%) and 2216 deaths among 29 011 magnesium-treated patients (7.64%), producing a nonsignificant mortality hazard of 4.0 ± 2.2 per 1000 patients treated. Indeed, the ISIS-4 investigators found no significant benefit of magnesium in any subgroup analyzed.

Several plausible explanations merit consideration in the attempt to resolve this apparent discord. Like nitrates, magnesium may be efficacious when given in the absence of other major therapies (as in the early, small trials) but may show no benefit in patients managed in a contemporary manner (as in ISIS-4). However, the use of meta-analysis to resolve differences between early and later trials of magnesium therapy has an additional layer of complexity, because the two most commonly used methods of meta-analysis, the "fixed-effects" model [2, 23] and the "random-effects" model [24], resulted in different summary estimates of effect. Specifically, the fixed-effects model found that the odds ratio for the nine trials of magnesium was 1.02 (CI, 0.96 to 1.09), and the random-effects model found the odds ratio to be 0.69 (CI, 0.47 to 1.02) [25] (Figure 2). Thus, despite combining results from an identical series of studies, one meta-analytic technique showed no evidence of efficacy, and another supported a potential benefit, albeit without statistical significance.

View larger version (36K): [in this window] [in a new window]   Figure 2. Magnesium therapy for acute myocardial infarction: trials and meta-analyses. The odds ratios of death and 95% Cls from seven smaller trials of magnesium therapy [11] and the LIMIT-2 [20] and ISIS-4 trials [15]. The two open bars are the results of the meta-analyses done using the random-effects model [25] and the fixed-effects model [15, 25]; they illustrate the discrepancy between these two forms of meta-analysis.

Although a formal discussion of the differences between fixed-effects and random-effects models is beyond the scope of this article, several issues have relevance for clinicians attempting to resolve these apparent discrepancies. In broad terms, a meta-analysis based on a fixed-effects model assumes that the trials being combined are derived from populations that are similar both to each other and to the population of interest to the reader. It also assumes that differences among trials are caused primarily by experimental error or within-study variability [3]. Further, the fixed-effects model assigns weight to the composite studies on the basis of individual variance and thus tends to favor trials with large samples over trials with small samples. In contrast, the random-effects model assumes that the trials being combined are but a potential sample of all conceivable populations, and it directly attempts to account for between-study variability [24]. In contrast to the fixed-effects model, the random-effects model tends to assign somewhat less weight to large trials, and it typically results in wider Cls because it takes heterogeneity among studies into account.

It is important for the clinician to recognize that both meta-analytic models have strong proponents within the statistical community [26-28] and that discrepancies between outcomes based on these models are relatively rare. Indeed, when there is minimal heterogeneity between studies, the random-effects and fixed-effects models tend to return similar estimates of effect [29]. At the same time, when considering the totality of evidence about magnesium therapy, it is relevant to recognize that neither meta-analytic technique resulted in a statistically significant demonstration of benefit, although the 95% Cls deriving from the random-effects model came close.

Regardless of issues related to which meta-analytic technique is used, an important remaining question is, Why did ISIS-4 differ from LIMIT-2? Both were relatively contemporary trials in which many patients received aspirin, thrombolytic therapy, and ß-blockers, unlike the trials included in the earlier meta-analysis. The LIMIT-2 investigators raised the possibility that magnesium may limit reperfusion injury and thus should be given before thrombolysis or direct angioplasty [30]. This hypothesis is supported by some animal experiments [31] and by the finding that the incidence of congestive heart failure was reduced in the group that received magnesium in LIMIT-2, which suggests the possibility of myocardial salvage. Although magnesium therapy was not shown to have any benefit in any subgroup of ISIS-4 [15], including those randomized within several hours and those at highest absolute risk, the specific hypothesis of limiting reperfusion injury was not directly tested, because the protocol specified that magnesium infusion be started after about the first hour of thrombolytic infusion [32]. An important limitation of the ISIS-4 protocol was that the actual time of magnesium initiation was unknown. However, reperfusion was not likely in either the earlier studies of magnesium that constituted the original meta-analysis or in a more recent study of high-risk patients ineligible for thrombolytic therapy [33]. Further trials in which patients receive magnesium before thrombolysis or angioplasty have been advocated [34]. Other authors [25, 34] have suggested that the relative benefit of magnesium may increase with the risk level of the population studied. These are tenable hypotheses that may be tested in future trials.

The discordant results of ISIS-4 and LIMIT-2 underscore a further limitation of meta-analysis that is relevant to the clinician caring for individual patients. Although meta-analysis is a powerful statistical tool for combining data, it is less capable of resolving pathophysiologic issues. This may provide important explanations of between-trial differences. Further, meta-analysis may not fully convey the shortcomings of the material used, either in terms of biology or epidemiologic bias [35].

How Should Clinicians Use Meta-analyses?

The discordant results of large-scale randomized trials, such as GISSI-3 and ISIS-4, indicate that even meta-analyses that meet high standards of statistical quality can be misleading. Particularly noteworthy are the apparent effects of changes in medical practice over time and the resulting differences in study populations, on the one hand, and the further bias introduced when only "appropriate" studies are selected for a contemporary meta-analysis, on the other. Discordance in the examples reviewed here is similar to that seen recently in a review of the perinatal literature [17].

Probably the most important caveat in interpreting meta-analytic results is that the findings must be considered primarily hypothesis-generating rather than hypothesis-testing. Strong recommendations for treatment should not be made on the basis of even the most promising meta-analysis in the absence of a sufficiently large and well-designed trial. At the same time, data from large trials are not necessarily definitive for patients who do not fit inclusion criteria or for drug regimens that differ importantly in dose, route, or administration. As long as meta-analysis is viewed as a prelude to or a rationale for a controlled, randomized trial, then the different results from megatrials and meta-analyses can be seen less as discrepancies and more as the outcome of an experiment whose results (produced by the large trial) differ from the hypothesis (produced by the meta-analysis). Clinical decisions should ultimately be made on the basis of a considered review of the totality of evidence from pathophysiologic principles, basic laboratory research, meta-analyses, and individual trials.

Dr. Ridker: Brigham and Women's Hospital, 75 Francis Street, Boston, MA 02115.