Using Numerical Results from Systematic Reviews in Clinical Practice

Making Sense of the Numerical Results of Clinical Studies

Although the results of clinical studies can be expressed in intuitively meaningful ways, such results do not always easily translate into clinical decision making. For example, results are frequently expressed in terms of risk, which is an expression of the frequency of a given outcome. (Risks are probabilities, which can vary between 0.0 and 1.0. A probability of 0.0 means that the event will never happen, and a probability of 1.0 means that it always happens.)

Consider a hypothetical study of the recurrence of migraine headaches in a control group receiving placebo and a treatment group receiving a new antimigraine preparation, drug M (a secondary prevention trial). Suppose that at the end of the trial, migraines recurred in 30% of the control group (the risk for recurrence was 0.30) but in only 5% of the drug M group (risk of 0.05) (Table 1).

The outcomes of the study are clear enough for the two groups when they are examined separately. But clinicians and patients are more interested in the comparative results, that is, the outcome in one group relative to the outcome in the other group. This overall (comparative) result can be expressed in various ways. For example, the relative risk, which is the risk in the treatment group relative to that in the control group, is simply the ratio of the risks in the two groups. In other words, relative risk is the risk in the treatment group divided by that in the control group, 0.05 ÷ 0.30, or 0.17. The comparison can also be expressed as the reduction in relative risk, which is the ratio between the decrease in risk (in the treatment group) and the risk in the control group, 0.25 ÷ 0.30, or 0.83 (Table 1). (The relative risk reduction can also be calculated as 1 −relative risk).

Although the clinical meaning of relative risk (and relative risk reduction) is reasonably clear, relative risk has the distinct disadvantage that a given value (for example, 0.17) is the same whether the risk with treatment decreases from 0.80 to 0.14, from 0.30 to 0.05, from 0.001 to 0.00017, and so forth. The clinical implications of these changes clearly differ from one another enormously and depend on the specific disease and intervention. An important alternate expression of comparative results, therefore, is the absolute risk reduction. Absolute risk reduction is determined by subtracting the risk in one group from the risk in the other (for example, the risk in the treatment group is subtracted from the risk in the placebo group). In the case of our migraine study, the absolute risk reduction would be 0.30 −0.05, which equals 0.25, or 25 percentage points. In contrast, for a study in which the risk decreased from 0.001 to 0.00017, the absolute risk reduction would be only 0.00083, or 0.083 percentage points, which is a trivial change in comparison (Table 1).

This arithmetic emphasizes the difficulty of expressing the results of clinical studies in meaningful ways. Relative risk and relative risk reduction clearly give a quantitative sense of the effects of an intervention in proportional terms but provide no clue about the size of an effect on an absolute scale. In contrast, although it tells less about proportional effects, absolute risk says a great deal about whether an effect is likely to be clinically meaningful. Despite this benefit, even absolute risk is problematic because it is a dimensionless, abstract number; that is, it lacks a direct connection with the clinical situation in which the patient and physician exist. However, another way of expressing clinical research results can provide that clinical link: the number needed to treat (NNT).

Number Needed To Treat

The NNT for a given therapy is simply the reciprocal of the absolute risk reduction for that treatment [1, 2]. In the case of our hypothetical migraine study (in which risk decreased from 0.30 without treatment with drug M to 0.05 with treatment with drug M, for a relative risk of 0.17, a relative risk reduction of 0.83, and an absolute risk reduction of 0.25), the NNT would be 1 ÷ 0.25, or 4. In concrete clinical terms, an NNT of 4 means that you would need to treat four patients with drug M to prevent migraine from recurring in one patient. To emphasize the difference between the concepts embodied in NNT and relative risk, recall the various situations mentioned above, in all of which the relative risk was 0.17 but in which the absolute risk decreased from 0.80 to 0.14 in one case and from 0.001 to 0.00017 in another. Note that the corresponding NNTs in these two other cases are 1.5 and 1204, respectively: that is, you would need to treat 1.5 and 1204 patients to obtain a therapeutic result in these two situations compared with 4 patients with drug M (Table 1).

The NNT can be calculated easily and kept as a single numerical reminder of the effectiveness (or, as we will see, the potential for harm) of a particular therapy. As we suggested, the NNT has the crucial advantage of direct applicability to clinical practice because it shows the effort that is required to achieve a particular therapeutic target. The NNT has the additional advantage that it can be applied to any beneficial outcome or any adverse event (when it becomes the number needed to harm [NNH]). The concept of NNT always refers to a comparison group (in which patients receive placebo, no treatment, or some other treatment), a particular treatment outcome, and a defined period of treatment. In other words, the NNT is the number of patients that you will need to treat with drug or treatment A to achieve an improvement in outcome compared with drug or treatment B for a treatment period of C weeks (or other unit of time). To be fully specified, NNT and NNH must always specify the comparator, the therapeutic outcome, and the duration of treatment that is necessary to achieve the outcome.

Important Qualities of the Number Needed To Treat

The NNT is treatment specific. It describes the difference between treatment and control in achieving a particular clinical outcome. Table 2 shows NNTs from a selection of systematic reviews and large randomized, controlled trials.

A very small NNT (that is, one that approaches 1) means that a favorable outcome occurs in nearly every patient who receives the treatment and in few patients in a comparison group. Although NNTs close to 1 are theoretically possible, they are almost never found in practice. However, small NNTs do occur in some therapeutic trials, such as those comparing antibiotics with placebo in the eradication of Helicobacter pylori infection or those examining the use of insecticide for head lice (Table 2). An NNT of 2 or 3 indicates that a treatment is quite effective. In contrast, such prophylactic interventions as adding aspirin to streptokinase to reduce 5-week vascular mortality rates after myocardial infarction may have NNTs as high as 20 to 40 and still be considered clinically effective.

Limitations of the Number Needed To Treat

Although NNTs are powerful instruments for interpreting clinical effects, they also have important limitations. First, an NNT is generally expressed as a single number, which is known as its point estimate. As with all experimental measurements, however, the true value of the NNT can be higher or lower than the point estimate determined through clinical studies. The 95% CIs of the NNT are useful in this regard because they provide an indication that, 19 times out of 20, the true value of the NNT falls within the specified range. An NNT with an infinite CI is only a point estimate; it includes the possibility of no benefit or harm. Such a point estimate may still have clinical importance as a benchmark until further data permit the determination of a finite CI, but clinical decisions must take this large degree of uncertainty into account.

Second, it is inappropriate to compare NNTs across disease conditions, particularly when the outcomes of interest differ. For example, an NNT of 30 for preventing deep venous thrombosis may be valued differently from an NNT of 30 for preventing a disabling stroke or for preventing death. The concept expressed by the NNT is thus one of frequency, not of utility; its numerical value is a function of the disease, the intervention, and the outcome. If we have NNTs for different interventions for the same condition (and severity) with the same outcome, then, and only then, is it appropriate to directly compare NNTs.

Third, NNTs are not fixed quantities. The NNT for a specified intervention in an individual patient depends not only on the nature of the treatment but also on the risk at baseline (that is, the probability at baseline that the patient being considered will experience the outcome of interest). Because that risk may not be the same for all patients, an NNT that is provided by the literature may have to be adjusted to compensate for your patient's risk at baseline, as described below. Moreover, the concept of NNT assumes that a given intervention produces the same relative risk reduction whether the patient's risk at baseline is low, intermediate, or high. This assumption may not always hold because, for example, a disease may be more difficult to treat when it is severe than when it is mild.

Finally, an NNT is always based on an outcome for a specified period. Imagine a disease that is treated by one injection or by regular daily tablets; the NNTs for the two treatments cannot be directly compared. Only when the outcome is the same and is measured during the same period is a comparison valid.

How Should Numbers Needed To Treat Derived from Systematic Reviews Be Used?

The distinction between therapy and prophylaxis is not always clear (for example, drugs for the treatment of hypertension). Because NNTs may be used differently in the two circumstances, however, it is often useful to distinguish therapy from prophylaxis. Thus, in situations that call for therapeutic intervention (treatment), some form of therapy will almost always be used, and the key issue therefore is the relative effectiveness of different interventions. For prophylaxis we more often have the choice of doing nothing; the issue then becomes a decision of whether doing something to prevent a bad outcome will be more successful than doing nothing. In contrast, in the case of treatment, the therapeutic equation for most patients consists of weighing the risks and benefits for each of the possible treatments. Under most circumstances, the equation for prophylaxis also includes the possibility of harm without benefit for a considerable number of the patients. For simplicity, therefore, we will separate treatment and prophylaxis and take a few examples from each.

Treatment

The NNT is particularly useful for treatments if several treatments are assessed for the same outcome measure in patients with similar conditions. Using the NNTs, we can rank these treatments relative to one another; this ranking is particularly helpful in making a choice on the basis of effectiveness. However, the resulting NNT league tables are not decision-making aids themselves because NNTs need to be balanced against adverse events; costs; and patient characteristics, expectations, and preferences. It is also important to keep in mind that favorable outcomes can occur without treatment and that the frequency at which this happens affects the NNT.

An example of the relative ranking of treatments can be seen in a comparison of studies of subcutaneous sumatriptan compared with placebo (NNT, 2.0) and oral sumatriptan compared with placebo (NNT, 2.6) for the relief (at 2 hours) of migraine headaches (Table 2). Because subcutaneous sumatriptan is more expensive than oral sumatriptan and the NNTs for the two treatments are similar, patient preference may be the deciding factor in choosing between formulations. An appropriate prescription for a woman in her mid-30s who has relatively frequent headaches and a high-powered position that involves a considerable amount of travel might be subcutaneous sumatriptan, but a retired biochemist who is troubled only by an occasional migraine might be more comfortable with oral sumatriptan. Knowledge about relative effectiveness can be accumulated as additional evidence appears, often from large randomized, controlled trials. If several studies show that aspirin plus metoclopramide for migraine had an NNT of 3 [21], for example, patients and clinicians might elect to change to this alternate therapy because of its lower cost and similar effectiveness when compared with the other two.

Another example of NNT ranking can be seen in reviews of treatments of diabetic neuropathy. Painful diabetic neuropathy affects about 3% of all diabetic patients after 20 years with diabetes. Four systematic reviews of drug treatments have used different approaches (Table 3).

The NNTs for antidepressant agents as a class (NNT, 2.5) were similar to those for anticonvulsant agents as a class (NNT, 2.9) in diabetic neuropathy (Table 3), but these systematic reviews do not tell us which individual drug was best in either class. Moreover, although the NNT for capsaicin was higher (NNT, 4.2), the overlap of the CIs for the NNTs of all three treatments suggests that we do not have definitive information with which to decide whether capsaicin is the least effective. We may, however, be prepared to make a judgment about whether the effectiveness as determined by the physician (the outcome measure used in the studies of capsaicin) is better or worse than pain relief of more than 50% as judged by the patient (the outcome measure used for the other two drug classes) (Table 2). Physician judgments are less sensitive than patient scoring [26]. For minor and major harm, no data were available on capsaicin. For minor harm (adverse effects) and major harm (withdrawal from the study because of drug-related toxicity), however, we know that anticonvulsant agents and antidepressant agents carry the same risk.

Choice of treatment for an individual patient depends on several issues. For example, this choice may be determined primarily by whether any of these drugs is licensed for the treatment of diabetic neuropathy (an external rule or constraint), by familiarity with a particular drug (physician knowledge and experience), by patient idiosyncrasy (patient factors), and so on. The point is that systematic reviews can provide valuable information that helps patient and physician to know with reasonable assurance what to expect from treatment.

Prophylaxis

With prophylaxis, the issue is the risk for an event occurring without prophylaxis compared with the risk with prophylaxis. Whether the medical condition is of major public health importance, such as heart attack or stroke, or less threatening, such as animal bites and the risk for subsequent infection, more people at risk will actually be unaffected than affected. The NNTs for prophylaxis tell us about the effectiveness for a population but are more difficult to use when deciding how to manage an individual patient.

As is the case for therapeutic interventions, part of the process of using information from systematic reviews of prophylaxis is to assess the risk at baseline (the risk for a bad outcome without treatment) for a particular patient, but this assessment is even more important in prophylaxis because a very low risk for a bad outcome at baseline makes prophylaxis difficult to justify. We must sometimes make that judgment ourselves and subsequently adjust risks and balance benefits and potential harms on the basis of experience, although we can often use evidence from other sources. In assessing the risk for gastrointestinal bleeding from nonsteroidal anti-inflammatory drugs, for example, a large randomized study [19] tells us that elderly persons who have a history of peptic ulcer, gastrointestinal disease, or heart disease are at the highest risk. The decision of whether to use prophylactic gastric protection will be guided by this information. Figure 1 shows some of the issues that are involved in making choices in prophylaxis.

An example for prophylaxis can be seen by a woman presenting to your office with a dog bite. Because she has been receiving long-term, moderately high-dose systemic steroid therapy for inflammatory bowel disease, you strongly suspect that the patient is immunocompromised. You are therefore concerned that she may be at increased risk for infection from the bite wound. The question is whether to give the patient prophylactic antibiotics to prevent such an infection. We know from a quantitative systematic review of randomized, controlled trials that has studied this question that evidence of benefit exists, with an overall NNT of 16 (Table 1).

How can this information be applied to your patient? Because she is immunocompromised, the patient's risk for developing an infection if she is not treated with antibiotics (sometimes referred to as the patient's expected event rate [3]) is considerably higher than that of the patients who were not immunocompromised in the systematic review. The patient's expected event rate might be estimated to be about five times greater than the 16% average rate of infection in the review (although the risk varied between 3% and 46% in individual studies). Assuming that the relative risk reduction is the same for high and low untreated risk, the estimated NNT that corresponds to the patient's estimated event rate is 16 ÷ 5, or 3. Thus, although antibiotic prophylaxis against subsequent infection in dog bites may not be worthwhile in all patients (NNT of 16 for patients who were not immunocompromised), it may be appropriate for this particular patient (NNT, 3). As an aside, if infection rates from dog bites in our area were much higher than the 16% in the review and approached the highest value that was found among the individual studies (50%), then we might be likely to give all patients prophylaxis with antibiotics.

Adverse and Other Events

The concepts that are captured by the NNT can also be used to express adverse events such as toxicity, side effects, or other harms. For minor adverse effects that are reported in randomized clinical trials, the NNH can be calculated in much the same way as the NNT. When the incidence of adverse events is low, it is likely that meaningful CIs will not be available (that is, the CIs may be infinite); therefore, only point estimates of harm will be available. Major harm may best be identified in randomized clinical trials through intervention-related withdrawal from the study; the NNH can be calculated from those numbers. Precise estimates of major harm often require a much wider literature search to find case reports or series, partly because these events are uncommon and partly because investigators may not report them in the full study, if they report them at all. The absence of information on adverse events in systematic reviews reduces the usefulness of such reviews (as in the case of topical capsaicin in Table 3).

Systematic reviews may also consider other consequences of treatment that may or may not be defined as adverse. A systematic review of the influence of epidural anesthesia during labor [27], for example, asked this question: If a woman is given epidural anesthesia, how much higher is her risk for having a cesarean section? In that review, a consistent increase in the rate of cesarean sections was noted in women who had epidural anesthesia. Sixteen percent of the women who had epidural anesthesia underwent cesarean sections compared with 6% of the women who did not have epidural anesthesia. The absolute risk increase was 10%, the relative risk increase was 161%, the NNT was 10 (CI, 8.4 to 13), and the odds ratio (see below) was 2.6 (CI, 2.1 to 3.2). This means that for every 10 women in labor who are given epidural analgesia, 1 will have a cesarean section who otherwise would not have had the operation if she had received another form of analgesia. The NNT of 10 provides a figure that can be used by women and their care-givers in making choices about their labor.

Calculating Numbers Needed To Treat If They Are Not Provided

For statistical reasons, event rates in two groups are often compared in terms of odds ratios rather than relative risks (or absolute risk reductions). Thus, whereas the risk for an event (probability) is expressed relative to a total universe of fixed size (for example, when 22 events occur in a population of 100 persons, the risk for that event is 0.22), the odds of that same event in that same population are calculated as the number of events relative to the number of non-events (for example, 22 to 78, or 0.28). An odds ratio, then, is simply the odds of an event in a treatment group divided by the odds of the event in the comparison group. If a quantitative systematic review produces odds ratios but no NNTs, the NNT can be derived from the data in Figure 2.

The easiest way to use Figure 2 is first to choose the column nearest the published odds ratio and the row closest to the event rate expected and then to read the corresponding NNT. Note that the odds ratios in the left section of Figure 2 are less than 1.0, meaning that the outcome of interest in the active treatment group is less common than in the comparison group; this is the situation in prophylaxis (in which the outcome is onset, recurrence, or worsening of disease). In contrast, the odds ratios in the right section are greater than 1.0, meaning that the outcome of interest is more common in the treatment group; this is the usual situation in studies of disease treatment (in which the outcome is cure, remission, or control of disease).

Figure 2 can also be used to determine how different values for event rate affect the NNT at a given odds ratio. Thus, if the rate of infection from dog bites in our area was 50%, then the NNT declines to 7 instead of 16 at an event rate of 16%. In such circumstances, as noted above, we might wish to use prophylactic antibiotics even for patients who are not immunocompromised.

As another example, the risk for cesarean section after epidural anesthesia, as noted above, has an odds ratio of 2.6; the event rate without epidural anesthesia is 6%. Using the odds ratios column of 2.5 and the event rate rows of 0.05 and 0.01 in Figure 2 yields an NNT somewhere between 9 and 15. This is close to the calculated figure of 10.

Odds ratios should be interpreted with caution when particular outcomes occur commonly, as in treatments for disease; odds ratios may then overestimate the effect of a treatment. Odds ratios are therefore likely to be replaced by relative risk reduction because relative risk reduction is more robust when event rates are high [3, 29]. If relative risk reduction is provided in a review, NNT can be estimated from a useful nomogram [30].

Variation in Occurrence of Events

The incidence of events in a comparison group can and does vary, often widely, from study to study. For example, in trials of droperidol to prevent vomiting after surgery for correction of strabismus, the incidence of postoperative vomiting varied enormously [13] (Table 2). In some trials, almost no postoperative vomiting occurred; in others, the incidence was greater than 50% with the same operation and nearly identical anesthetics. Wide variation in event rates occurs with treatment and with prophylaxis. In six trials of natural surfactant for preterm infants, the event rate for bronchopulmonary dysplasia was 24% to 69% [31]. Under other circumstances, the variation in event rate may be narrower. For example, the rate of ulcer healing without antibiotic treatment ranged from 0% to 17% in 11 randomized, controlled trials that studied the therapeutic effect of eradicating H. pylori infection with antibiotics on ulcer healing [5].

The effectiveness of prophylaxis and treatment depends on the risk for the event without the active intervention. Thus, if patients do not vomit at baseline, then prophylaxis is not necessary; if most of them vomit, then prophylaxis may be particularly useful. If patients rarely recover from a disease without treatment, then treatment may be highly appropriate; if most patients recover on their own, then treatment may or may not be useful. The patient's expected event rate thus becomes important for the therapeutic or preventive decision, even when an intervention is proven to be effective.

Comments

A systematic review that is done properly can locate most of the useful information that has been published on a medical intervention. Such a compendium of information provides much more power than is often available from single trials because trials, particularly those that evaluate treatments, are often conducted with few patients in the experimental and comparison groups. During use of the results from systematic reviews, however, it is important to be able to shift from the numbers that are generally used to express the amount of benefit or harm from an intervention to a number that captures the effort that is necessary to achieve that benefit (or avoid that harm) in a given patient. Distilling the results of systematic reviews into, in effect, one number (the NNT or NNH) provides a measure of that effort and is therefore a clinically relevant approach. Physicians and patients can use this approach to rapidly, quantitatively, and accurately estimate the amount of benefit and any accompanying harm for a given intervention. These calculations are simple to remember and use in personal or institutional practice; they should help us to make the best possible clinical decisions with our patients.

Like all research results, however, NNTs are only one element of decision making and need to be integrated with patient preferences, caregiver experience and judgment, and local constraints and conditions. It is also worth noting that when clinicians and policymakers were presented with research results in different formats (NNT and absolute and relative risk reduction, among others), they made more conservative decisions when they received treatment effects expressed as NNTs than when they received them as relative risk reductions or absolute risk reductions [32-34]. (Table 4)

Glossary

Relative risk: Risk for achieving an event (with treatment) or preventing an event (with prophylaxis) in the treatment group relative to that in the control group.

Relative risk reduction or increase: Increase in events with treatment compared with control (treatment) or reduction in events with treatment compared with control (prophylaxis); this number is often expressed as a percentage.

Absolute risk reduction: Difference in event rates for two groups, usually treatment and control.

Number needed to treat: Number of persons who must be treated for a given period to achieve an event (treatment) or to prevent an event (prophylaxis). The NNT is the reciprocal of the absolute risk reduction.

For more information, see [1-3].