As they must with all powerful tools, users need to understand and respect the characteristics of predictive models to use them effectively and avoid the most serious pitfalls associated with their application. Statistical models whose outcomes are expressed as probabilities are called probability models [1]. We consider here some of their characteristics—how they work; when, where, and how to apply them and how not to apply them; and their major strengths and limitations.

Predictions That Use Several Variables

Your patient is a 45-year-old man who has recently had "definitive" surgical removal of a malignant melanoma. Both you and the patient would like to know the patient's chances of being alive, with or without recurrent melanoma, 10 years after such surgery. One approach to solving this problem would be simply to use the initial diagnosis of melanoma alone to predict survival. To do this, you would need to find a study of a group of patients whose melanoma had been treated surgically and who had then been followed for 10 years, to see what proportion of those patients died of the disease. In a study published in this issue [2], Schuchter and colleagues did just that. They found that 22% of a group of 488 patients had died of recurrent melanoma by the end of a 10-year period (78% crude overall survival).

Unfortunately, the amount of predictive information provided purely by knowing the diagnosis of nonmetastatic melanoma is limited. Fortunately, patients rarely bear simple diagnoses; they present with rich and complex arrays of historical, physical, and laboratory findings. But unless clinical prediction instruments use a substantial amount of this information, predictions about individual patients lack power and accuracy.

Schuchter and colleagues developed a probability model that uses four categories of thickness of the resected primary tumor to predict the probability of 10-year survival after surgery (the thickness-alone model). This is the statistical analogue of the qualitative clinical approach that uses tumor thickness to make an informal prognosis. This model replaces the single observed percentage (78% crude overall survival) with four estimated probabilities of surviving for 10 years: 96% for patients with tumors less than 0.76 mm, 83% for patients with tumors between 0.76 and 1.69 mm, 59% for patients with tumors between 1.70 and 3.60 mm, and 29% for patients with tumors greater than 3.60 mm Table 4 in the study by Schuchter and colleagues [2]). In effect, the thickness-alone model describes how survival probability "changes form in different [thickness] subgroups of a population" [3].

Schuchter and colleagues developed a four-variable probability model with which to predict 10-year survival even more accurately and selectively. In this model, tumor thickness, site of the primary tumor at the time of definitive surgical therapy, and the age and sex of the patient were all found to be independent predictors of survival 10 years after surgery. This model is called a multivariable model because it estimates the joint effect of multiple input (independent) variables on the study outcome. When an outcome has two categories (for example, dead or alive at 10 years), it is called binary, and (binary) logistic regression is the type of statistical model that is generally used to predict the probabilities for the two outcomes. The thickness-alone model and the four-variable model are both logistic regression models.

At their best, multivariable statistical models can provide more accurate predictions (for example, prognoses) than can expert clinicians or traditional prognostic algorithms [4] because the multiple independent variables do, in fact, use multiple independent sources of information about the outcome. Why, then, do physicians use probability models so infrequently when making predictions at the bedside for individual patients? One major obstacle is the complexity of the algebraic equations used to represent probability models (see, for example, the four-variable multivariable logistic regression equation in Table 1 of the article by Schuchter and colleagues [2]) coupled with the need to do laborious computations. (The current wide availability of portable computers should help overcome this problem, and commercial software programs are available that use probability models to predict certain medical outcomes, such as the survival of patients who require intensive care.)

A simpler and more direct approach is to construct a table showing the outcome probabilities calculated for every possible combination of findings—and thus for every type of patient—addressed by a particular probability model. This representation of the model makes it possible to directly look up the probability for your specific patient. Schuchter and colleagues present such a table Table 3 in their article [2], adapted here as Table 1), which shows all of the predictions made by the four-variable model; that is, it contains estimates of the probability of 10-year survival for patients with melanoma who have any one of the 32 possible combinations of the four variables. Although the table obviously does not capture all of the complexity of a real patient, it does allow predictions to be considerably tailored to the individual patient.

Seven Things You Need To Know before Using a Probability Model

Statistical models are abstractions, but they are constructed by using concrete, empirical data obtained from carefully chosen study groups (samples). For example, data from 488 patients were used to construct the four-variable model of 10-year survival after the removal of a primary cutaneous melanoma [2]. The challenge to clinicians is in knowing how to move from the abstract back to the concrete, that is, knowing when and how to apply a particular probability model to a particular group or individual patient.

Suppose, now, that you find an article such as that by Schuchter and colleagues containing a multivariable model for survival in patients with melanoma. What conditions does a prediction model need to meet before you can consider using it for your particular patient? And how do you actually use the model to make predictions for this patient?

We suggest that answering the following seven questions will provide a systematic, practical approach to the problem. The first five questions help the clinician judge whether a specific model is suitable for a particular patient; the last two questions address how well the model itself works. All seven questions are important; the order in which they are applied is a matter of convenience.

Questions about the Suitability of a Specific Model for a Particular Patient

1. Would Your Patient Have Been Eligible To Enter the Study That Led to the Model?

To use the model described by Schuchter and colleagues [2], you must first decide whether your patient with melanoma has the same type and stage of disease as the patients entered into the study of Schuchter and colleagues: primary clinical American Joint Commission on Cancer stage I or stage II malignant melanoma [2]. If this is the case, you need to determine whether the necessary ancillary information, such as data related to the exclusion criteria of the study, is available for your patient and does not disqualify him or her. If, for example, your patient has evident metastatic disease beyond the primary site at presentation or lacks negative margins after wide excision of the primary tumor—conditions that were used to exclude patients when the model was constructed [2]—the model is not applicable to your patient. Similarly, it is inappropriate to apply the model to patients who have comorbid conditions that substantially reduce life expectancy, because patients with melanoma who died of causes other than melanoma before 10 years had elapsed were not used to develop the model.

2. Does the Outcome Variable in the Statistical Model Reflect the Clinical Outcome You Want To Predict?

Is the outcome used by the model—10-year survival—similar to the outcome you want to predict? If "long-term survival" (that is, survival with or without melanoma for longer than 10 years) is your primary concern, the model presumably provides an adequate approximation. This is because the probability of long-term survival is not much lower than the probability of 10-year survival (death from melanoma after 10 years does occur, but only infrequently). If you need to predict the probability of survival at many times for a patient after resection of the primary tumor, you would not be able to apply this model. Instead, you would need to use one of the survival regression methods (such as Cox regression) specifically designed for that purpose [4, 5].

3. Are Values for All Independent (Input) Variables Available for Your Patient Where and When You Need To Use the Model?

The values of the four independent variables used in the model of Schuchter and colleagues (that is, the age and sex of the patient and the body site and thickness category of the primary tumor [2]) are available for almost all patients with melanoma. It is likely, therefore, that you will have the data that you need to apply the model to your patient (and to other patients with melanoma) at the time at which you need to predict survival.

In contrast, an earlier model from Clark and coworkers [6] predicted survival for patients with melanoma (for 8 years rather than 10 years) by using six independent variables: mitotic rate, tumor infiltrating lymphocytes, histologic regression, tumor thickness, body site of the tumor, and sex of the patient Table s6 and 7 in their article [6]). Unfortunately, most clinicians found that data for at least one of the first three input variables were not routinely available for patients in their practices who had melanoma [2]. Because all six variables were required to compute the survival probability estimate, the result was that the six-variable model could only rarely be used to predict the probability of 8-year survival. Schuchter and colleagues [2] therefore developed the more usable model containing four readily available clinical predictors: patient age and sex and the site and thickness of the tumor, as mentioned above.

The earlier and more elaborate model remains useful, however, because it describes the direction and strength of the associations between each of the six independent variables and 8-year survival from melanoma—this can be thought of as an "explanatory" use of the model. In contrast, the four-variable model is not as effective as the more elaborate model in providing an understanding of the underlying biological process of melanoma, because some clinically important variables independently associated with outcome were left out. The simpler model, however, is more useful for pragmatic, "predictive" purposes at the bedside. These examples underscore the point that models are serviceable simplifications of a complex reality, and no one model is best for all purposes. Thus, as in clinical reasoning generally, the level of completeness and precision required of a probability model depends on what the result will be used for. These subtle but critical aspects of probability models are captured in the apt summary phrase, "Models are to be used, but not to be believed." Although statistical models are commonly used for "explanatory" purposes [3, 7-12], we consider only their "predictive" use in individual patients [4-612, 13] from this point on.

4. Will the Outcome Probability (Estimated from the Model) Help in Prognosis, Choice of Treatment, or Other Aspects of Patient Care[14]?

Patients with surgically treated malignant melanomas whose prognosis is good may choose to forego certain highly toxic experimental treatments. Conversely, patients whose underlying prognosis for survival from melanoma is poor may choose to risk the increased toxicity of experimental treatment. In sum, an accurate, quantitative appraisal of survival prognosis can give the patient, the family, and the physician important information tools that would otherwise not be available. These tools can be used in making treatment decisions and may also help in other aspects of life planning.

5. Is the Uncertainty in the Predicted Outcome Probability Small Enough for That Estimate To Aid in Making a Specific Prognosis?

A probability model produces a single number, called a point estimate, that expresses the probability of the clinical state of interest for each patient with a given combination of clinical findings. What is usually not stated, however, is the degree of uncertainty in those point estimates. This particular kind of uncertainty arises because even the most representative study group (a "random sample") used to create a model can only partially represent the true "universe" of all such patients [13, 15].

Confidence intervals express the margin of error (uncertainty) in point estimates that arises from using study samples rather than all patients of interest. Even these important measures usually underestimate that error because study samples are rarely random samples [15]. Consider, for example, a 45-year-old man who had a primary melanoma on his right leg that was 4 mm thick. Two types of estimates of survival probability for this patient are found in the seventh row and second column of the probability table (Table 1). One is the 95% CI estimate (0.36 to 0.77), which is the statistically plausible range of probability estimates for this patient's 10-year survival [16]. The other is 0.58, the point estimate of the percentage of patients in this cell who are predicted (calculated using the model) to survive for 10 years. Although the point estimate is the best single-number estimate, it fails to provide any sense of the degree of uncertainty surrounding it. A rough estimate of that uncertainty is, in fact, provided by the 95% CI. Thus, all of the numbers between 0.36 and 0.77 are "possible values for the true [probability] that are reasonably consistent with the observed data" [17]. This 95% CI is so wide that the numbers at its opposite ends, 0.36 and 0.77, suggest two altogether different prognoses. Thus, although the point estimate of 0.58 appears to be "exact," it is too uncertain in this instance to be used clinically. Unfortunately, CIs are seldom included along with the point estimates of outcome probabilities. It should be obvious from this example, however, that using a single-number estimate without considering the uncertainty that surrounds it can be misleading [9, 15].

Questions about How Well a Particular Model "Works"

If any of the answers to questions 1 through 5 is "no," the model should not be used to predict survival for the patient in question. If the answers are all "yes," the model may be useful. However, at least two additional questions on how well the model actually "works"—the fit of the model to actual group data and its discriminating or predictive power—need to be answered before the model is used to make a prediction for an individual patient.

6. Are the Outcome Probabilities Estimated from the Model Sufficiently Close to the Outcomes Actually Observed in the Study Group (That Is, How Well Does the Model "Fit" the Data)?

Suppose, now, that you examine a 35-year-old woman who has a primary melanoma on her left arm that is 3.8 mm thick. If you looked up the survival probability of this patient by using the four-variable model for patients with this particular combination of clinical variables, you would obtain a value of 0.74 (95% CI, 0.53 to 0.87), which indicates a fair to good prognosis (row 7, column 1 of Table 1. Curiously, however, by looking at the fraction on the second line of the appropriate cell in Table 1 [column 1, row 7], you would also find that only 3 patients of the original group of 488 were similar to the patient with respect to age, sex, tumor site, and tumor thickness and that all 3 of these patients survived for 10 years (1.00 actual survival proportion); this indicates a much better prognosis than 0.74, or even 0.87. Why the discrepancy? And why not use 1.00, the survival actually observed, rather than the number calculated by using the model?

The explanation is that the 1.00 survival probability for patients with these particular characteristics is based on such a small number of patients—in this case, only three—a small part of the large group of patients used to develop the overall model. One of the many benefits of a "good-fitting" model, then, is that "the model's predicted values (here probabilities) smooth the data and provide improved estimates" of the true survival probabilities [18], particularly within small subgroups of patients. But does the four-variable model fit the observed data well?

Comparing the model's point estimate prediction of 10-year survival (the decimal number on the left) with the actual survival figure (the fraction on the second line) within each subgroup (cell) in Table 1 gives a general sense of how well the model's predictions "fit" the data. The differences between these two proportions in each of the 32 cells vary from very large (that is, a difference of 0.42 [1.00 (5/5) observed – 0.58 predicted survival probability in the seventh row, second column]) to very small (that is, a difference of 0.01 or 0.02 in the top row). The assessment of "goodness of fit" obtained by simply "eyeballing" the data is, of course, rough, but formal statistical tests are fortunately available for this purpose. Using the widely accepted Hosmer-Lemeshow "goodness-of-fit" test for logistic regression models [10], Schuchter and colleagues [2] found no evidence of an overall lack of fit of the four-variable model to the actual data. In general, one should be wary of a multivariable model for which no goodness-of-fit test is reported. Ideally, a statistician using relevant "regression diagnostics" should also be convinced that no pattern can be discerned in the differences between predicted and observed values before the fit of any particular model is accepted [9-11].

7. Is the Model More Accurate Than Both Chance and Traditional Methods of Prediction, and Was This Assessed by Using Follow-up Data?

Answering the second part of this question first, Schuchter and colleagues [2] used 10-year survival status, which was obtained by following each patient after tumor excision, to test the model's predictions. Because model predictions cannot be tested without follow-up data, only predicted probabilities obtained from models based on longitudinal data should be used to test the accuracy of a model's predictions, as Schuchter and colleagues did. Meaningful outcome probabilities cannot ordinarily be obtained from data taken from cross-sectional or case–control studies [7].

Even if a model's predictions fit the outcome data well, the model may have little power to discriminate between outcomes because the input variables may not be strongly associated with the outcome of interest. Thus, in addition to measuring a model's fit to the data, it is important to determine the model's prediction accuracy (or its counterpart, the prediction error). Using boxplots, Schuchter and colleagues showed that the probabilities of 10-year survival predicted by using the four-variable melanoma model are generally higher for those patients who ultimately lived than for those who died of melanoma within 10 years, as they should be Figure 1 in the article by Schuchter and colleagues [2]).

Another measure—called the concordance index, or c-index—provides a single-number summary of the prediction accuracy of the model, that is, of how well the model discriminates between patients who will die and patients who will be alive 10 years later [19]. More specifically, the c-index considers all pairs of study patients, one of whom survived for 10 years and one of whom died of melanoma. The index itself is the percentage of pairs for which the model correctly predicts a higher probability of survival for the patient who actually survived. The c-index can range from 0.0 (all incorrect predictions) through 0.5 (chance prediction) to 1.0 (all correct predictions). In the four-variable melanoma model, the c-index is 0.87, indicating that the model correctly assigns a higher survival probability to the actual survivor 87% of the time [19]. Because the entire 95% CI (0.84 to 0.91) for this c-index is greater than 0.5 (chance prediction), we can say that the model predicts statistically better than chance. Because the interval is so far above 0.5, we can also say that the model predicts well. Schuchter and colleagues [2] describe the ability of the four-variable model to discriminate outcomes in terms of a different measure, the area under the so-called receiver-operator characteristic (ROC) curve. For binary logistic regression models (such as the four-variable melanoma model), however, that measure equals the c-index [19, 20]. The c-index is also applicable to many other types of probability models (see Appendix).

The final question is whether the four-variable model predicts survival after surgery for melanoma more accurately (that is, with less prediction error) than the more traditional clinical method, represented here by the thickness-alone model. The c-index of 0.87 for the four-variable model is statistically greater than the 0.82 provided by the thickness-alone model, a difference that is not only statistically significant (P < 0.002) but also clinically significant [2]. Thus, in the group of 488 patients with which the model was developed, the four-variable model is better than the simpler thickness-alone model at distinguishing survivors from persons who will die of melanoma within 10 years.

As Mosteller and Tukey [21] point out, however,

Testing a (model) on the data that gave it birth is almost certain to overestimate performance, for the optimizing process that chose it from among many possible (models) will have made the greatest possible use of any and all idiosyncrasies of those particular data ... . As a result, the procedure will likely work better for those data (sometimes producing spurious associations) than almost any data that will arise in practice.

Efron and Tibshirani [22] and others [23] have distinguished between the "apparent prediction error" of a model, which is the error estimated in the development sample, and the larger "true prediction error," which is the error that occurs when the model is used in practice.

The true prediction error can be estimated in several ways, including through the use of procedures known as data-splitting, cross-validation, and bootstrapping [22, 24]. As an example of the first procedure, Schuchter and colleagues [2] split their entire available data set into two parts: 1) the first 488 patients with melanoma enrolled between 1972 and 1979 [the development sample, which they used to develop the model] and 2) the last 142 patients who entered the study between 1980 and 1981 (the validation or test sample, which they used to test the model). They found that the four-variable model developed from the patients enrolled between 1972 and 1980 predicted 10-year survival well for the patients enrolled in 1981 and 1982. The validation sample was then used to compare the error of the four-variable model in predicting 10-year survival outcome with that of the thickness-alone model. The area under the ROC curve was greater for the four-variable model in this second group as well, indicating that its advantage extends to the validation sample [2].

In the cross-validation and bootstrap procedures, many different hypothetical patient groups are created by repeated sampling from the entire original study group (for example, 630 patients [488 + 142] in the study of Schuchter and colleagues [2]). Separate probability models are then developed on each of these hypotheticals and tested against a reference (a patient or patient group) to estimate what the prediction error of the original model would be if it were actually used in a new patient group. Both cross-validation and bootstrapping, like data splitting, use data from a single study, but they make more efficient use of the study data; that is, they use all the data rather than a fraction of the data to develop the model [5, 25]. On the other hand, neither of these two methods allows us to see how the model would fare in a different, "real" group of patients a short time after its development, as does the data-splitting method used by Schuchter and colleagues [2].

None of these methods for estimating the prediction error of the model assures us of the model's generalizability to other groups of patients, such as those in other cultures, ethnic groups, or geographic regions. Thus, further corroboration of the 10-year survival model in different patient populations and settings is an important next step in evaluating the model's validity and usefulness, as Schuchter and colleagues point out. More generally, caution should be used when a model is applied to patients who differ substantially from those in the original study group, even if those patients would have satisfied the study entry requirements. For example, the model may not be as accurate for patients in settings in which the overall 10-year survival rate differs substantially from the 78% and the 75% observed in the development and validation samples, respectively [2].

Conclusions

Probability models are powerful tools with which to predict the probability of clinical states. They have not been widely used for making predictions in individual patients, but they can be used in this way; the resulting predictions would be greatly facilitated if the models were presented in tabular form. Single-number estimates (point estimates) of the probability of clinical states may lead users to underestimate the actual uncertainty hidden in those predicted probabilities. Because the 95% CI makes users aware of uncertainty, CIs for probability estimates should be included in the tabular display of the model.

More generally, even estimated CIs for outcome probabilities are simplifications of a complex reality and should not be regarded as "facts" or "truth." Rather, model-derived probabilities are abstractions that conceal several levels of uncertainty, and the width of the CI addresses only one of many potential sources of error in the probability estimate because even adequately tested models may have excluded other important predictors. Furthermore, as Horwitz and Ferleger [26] state, "most predictions have as their basic assumption ‘If current trends continue ... .’ Unfortunately they usually don't." For example, secular changes in treatment or in the course of a disease can reduce the applicability of even a well-fitted and validated model. In sum, validated probability models can provide clinically useful predictions, but those predictions are always uncertain to some degree. The more we understand the kinds of uncertainty, the more informed we will be about the application of these models to individual patients.

Given all of these complications, why should clinicians use these models? In our view, certain critical management decisions call for quantitative predictions of considerable accuracy, and probability models can provide such predictions. Indeed, some have suggested that "an understanding of probabilities ... enables patients and physicians to exert all possible influence over the outcome of an illness and to gain the satisfaction and the sense of control that comes from doing so" [27]. Equally important, perhaps, is the recognition that qualitative clinical predictions face similar problems with uncertainty but that those uncertainties are seldom explicitly confronted. In fact, most of the concerns addressed in this paper are not specific to prediction models. They are, rather, generic to prediction itself.

Appendix

The way a study outcome (dependent variable) is measured is one important determinant of the type of probability model that should be used for prediction. As noted, because the outcomes in the melanoma examples were binary (that is, dead or alive at 8 years and at 10 years), (binary) logistic regression was the type of probability model used to calculate outcome probabilities in those models. If the outcome variable has three or more categories, ordinal logistic regression is used when the categories are ordered and polytomous (multinomial) logistic regression is used when the categories are unordered [1, 9]. The results of Poisson regression models may be expressed as probabilities of events (for example, the probability that a patient with angina will have 0 or 1, 2, 3, 4 ... angina attacks during a week [1, 9, 13].

All of the probability models (including the types described above) used for prediction in individual patients can be evaluated using the "seven-question" approach. Moreover, the c-index "is a widely applicable measure of predictive discrimination, one that applies to ... (binary) outcomes, ordinal outcomes and censored time until event response variables (e.g. Cox regression)" [5].

Many other important issues in the use of multivariable models are not addressed in this essay, including the importance of minimal loss to follow-up, ordinal and continuous independent variables, testing for interactions between independent variables, checking for "overdispersion," collinearity of (independent) variables, handling outliers (influential observations), and violations of the assumptions underlying statistical models [5, 9, 11, 14, 28].

Dr. Davidoff: Annals of Internal Medicine, American College of Physicians, Sixth Street at Race, Philadelphia, PA 19106.