Predictions of Hospital Mortality Rates: A Comparison of Data Sources

Methods

Collection and Classification of Data Elements

Inpatient data for risk-adjustment models were obtained between January 1991 and December 1992 from 30 acute care hospitals in Cleveland, Ohio [15]. The 30 participating hospitals range in size from 82 to 899 beds. Twenty-four hospitals are private nonprofit institutions, 5 are affiliated with churches, and 1 receives public (county) support. Nine hospitals have medical school affiliations, 11 have accredited residency programs (9 medical and 2 osteopathic programs), 7 are certified trauma centers, and 10 perform open heart surgery [16].

Data were obtained for 46 769 adults (age ≥ 18 years) who were consecutively discharged from the 30 participating hospitals after medical treatment for acute myocardial infarction (6088 patients; mean age, 68.7 years), cerebrovascular accident (9061 patients; mean age, 72.9 years), congestive heart failure (18 864 patients; mean age, 73.6 years), or pneumonia (12 756 patients; mean age, 68.9 years). These medical conditions had been selected previously by the participating hospitals and their corporate customers as collectively representing the highest proportion of admissions for acute care, a major share of hospitalization costs, and the highest proportion of in-hospital deaths [15]. Patient eligibility was determined from specific ICD-9-CM principal diagnosis codes. These codes did not include patients who had surgery. (Codes used to identify patients with each medical condition are listed in Appendix A.)

Data elements were abstracted from patients' records by medical records personnel trained in data abstraction and were designated as administrative, laboratory, or clinical. Administrative data elements comprised demographic characteristics, sources of admission (that is, the route and external source), and diagnostic information derived from ICD-9-CM diagnosis codes. To obtain diagnostic information, the ICD-9-CM codes entered by the hospitals' professional coders into the medical record were transcribed by the data abstractors. These are the same codes that were submitted by the hospitals for reimbursement, and their accuracy is subject to regular audit by peer review organizations. Laboratory data elements included blood chemistry, hematologic, and arterial blood gas variables. Clinical data elements were the findings seen at hospital admission that are generally available only in patients' charts (for example, chest radiographic and electrocardiographic findings, mental status, and vital signs).

Development of Risk-Adjustment Models

For each diagnosis, stepwise logistic regression [17] was used to develop administrative, laboratory, and clinical risk-adjustment models. Because comorbid conditions are not always distinguishable from complications, both restricted and unrestricted administrative models were considered. All variables derived from administrative data were eligible for inclusion in the unrestricted administrative models. The restricted administrative models included only the variables (such as diabetes mellitus, cancer, and chronic renal failure) that were unlikely to be complications of care. Only administrative data elements were included in the two types of administrative models. The laboratory and clinical models contained both restricted administrative data and laboratory data. (Appendix B lists all data elements found in one or more of the final risk-adjustment models.)

Only variables that showed a univariate association with death were considered for inclusion in the stepwise logistic models. Each continuous variable was screened to determine whether its relation to mortality was approximately linear. If so, the variable was treated as continuous within a specified range. If the association was nonlinear, the variable was represented by a set of dichotomous variables that corresponded to different ranges of values.

We excluded patients for whom any vital sign was missing. Routine laboratory results (blood urea nitrogen, creatinine, glucose, and electrolyte levels and complete blood counts) were missing in 2.3% to 4.0% of the patients included. More specialized blood chemistry results (albumin, calcium, aspartate aminotransferase, lactate dehydrogenase, bilirubin, and alkaline phosphatase levels) were missing in 14.7% to 20.2% of included patients. Measurements of blood gases, creatinine phosphokinase levels, prothrombin time, and partial thromboplastin time were missing in 35.9% to 50.9% of included patients. Specialized laboratory tests are often omitted when physicians believe that the results of these tests would be normal or redundant or because critically ill patients have died. Differences among hospitals in the accuracy and completeness of data recording were assessed and were found to be very small.

To account for missing data on specific tests in specific conditions, mortality rates associated with ranges of observed laboratory values were compared with mortality rates for patients for whom data were missing; patients who did not have a given value or values received the values associated with the most similar mortality rate. For example, the mortality rate for the 12.5% of patients with acute myocardial infarction in whom albumin levels were not documented was 21.5%, whereas the overall mortality rate was 14.1%. Mortality rates in patients with acute myocardial infarction whose albumin levels were documented ranged from 42.3% for patients whose levels were less than 20 g/L to 6.1% for those whose levels were greater than 45 g/L. Therefore, missing albumin levels were imputed at 32 g/L, the midpoint of the range of albumin levels of patients whose mortality rate was closest to that of patients with no albumin values. This method was used for all continuous variables (that is, the alveolar-arterial oxygen gradient in myocardial infarction; albumin levels in myocardial infarction and congestive failure; alkaline phosphatase levels in all conditions; bilirubin levels in congestive heart failure; creatinine levels in myocardial infarction; and blood urea nitrogen levels in congestive heart failure, stroke, and pneumonia). For dichotomous variables whose presence was associated with increased risk, imputed values were almost always set to “absent.” More detailed information on the models and imputation can be obtained from the authors.

Each of the stepwise models was reviewed to ensure that the variables and their signs did not disagree with medical wisdom. Marginally statistically significant independent variables that described important known clinical risks were retained. Highly correlated variables were identified and eliminated.

Evaluation of Discriminatory Power

The discriminatory power of risk-adjustment models was evaluated by comparing c-statistics, which are equivalent to areas under receiver-operating characteristic (ROC) curves [18]. For all possible pairs of patients in which one lived and one died, the c-statistic is the proportion of times the patient who died had the higher predicted probability of dying. The Hosmer-Lemeshow statistic, with 8 degrees of freedom [19], was used to examine the models' goodness of fit. Because the value of the c-statistic is proportional to the sample size and because the samples in our study were large, the observed significance level for the c-statistic was considered to be too sensitive as an absolute measure of fit. Nevertheless, only one of the models had a chi-square value greater than 35. Paired comparisons of areas under ROC curves for models based on different sources of data were done using the method of Hanley and McNeil [20].

To compare patient-level mortality predictions that were obtained by using the clinical models with those obtained by using the alternate models, patients in each condition-specific group were ranked according to their predicted probability of dying, as determined by each model. For each model and condition, patients were then assigned by rank to 1 of 10 groups of equal size (deciles). For each patient, the absolute value of the difference in decile rank between the clinical model and each of the alternate models was calculated. For each condition, we calculated the percentage of patients who had decile rank differences of two or less and one or less. (Deciles that differed by two or more were usually associated with at least a twofold difference in mortality rates.) We also examined the distribution of the predicted probabilities of dying for each model to determine whether the models differed in terms of the distribution of large and small probability values.

To assess the potential effect of the data source on measured hospital performance, predicted mortality rates based on each of the models were calculated for each condition in each hospital. For the clinical models, SDs of the predicted rates were also computed. The difference between the mortality rates predicted by the clinical models and the mortality rates predicted by each of the alternate models was computed. For each condition, the percentage of hospitals with differences less than 1 SD and less than 2 SDs was computed.

We sought to determine whether ICD-9-CM codes in the unrestricted administrative model might represent inpatient complications. To do this, we examined abstracted clinical data to identify patients for whom all coded conditions were present at hospital admission. The remaining patients (those without clinical evidence of the coded illness or illnesses at the time of admission) were grouped into deciles of risk, as predicted by the unrestricted administrative models and by the clinical models. For patients in whom the difference between the two sets of predictions was three or more deciles, we examined the incidence of ICD-9-CM codes that might represent inpatient complications rather than conditions present at hospital admission [13].

Results

The overall observed mortality rate was 9.2%. The mortality rates for patients with each condition were 13.3% for patients who had acute myocardial infarction, 6.7% for patients who had congestive heart failure, 10.8% for patients who had a cardiovascular accident, and 9.7% for patients who had pneumonia.

The effectiveness of the models in predicting death differed by data source rather than by medical condition. Table 1 lists the areas under the ROC curves (c-statistics) for each of the 16 predictive models. The use of restricted administrative data alone gave the least accurate mortality predictions (average c-statistic, 0.75). Both the unrestricted administrative data and the combination of restricted administrative, laboratory, and clinical data gave the best mortality predictions (average c-statistic, 0.87; P < 0.001 compared with restricted administrative data alone). Restricted administrative and laboratory data together yielded mortality predictions that were substantially higher than those obtained by using restricted administrative data alone (average c-statistic, 0.86; P < 0.001) and only slightly, although statistically significantly (P < 0.001), less than mortality predictions achieved by using abstracted clinical data or the unrestricted administrative data alone.

Table 2 shows, for each condition, the percentage of patients who had decile ranks similar to those of the clinical models. For all conditions, the laboratory models produced the greatest agreement. Overall, 93% of the patients had decile ranks assigned by the laboratory models that were the same or that differed by only one or two deciles from the ranks assigned by the clinical models. Only 75% of rankings assigned by the unrestricted administrative models differed by two or fewer deciles from the rankings assigned by the clinical models. For the restricted administrative models, only 72% of rankings differed by two or fewer deciles from the rankings assigned by the clinical models.

When we examined the percentage of patients whose predicted values were greater than 0.4, we found that only 1.6% of the probabilities obtained by using the restricted administrative models exceeded this value, compared with 5.5% of the probabilities obtained by using the clinical models, 5.0% of the probabilities obtained by using the laboratory models, and 4.9% of the probabilities obtained by using the unrestricted administrative models.

Table 3 shows the differences in predicted hospital mortality rates between the clinical models and each of the alternate models. Overall, 95% of the differences between the mortality rates predicted by using the laboratory models and the clinical models were within 1 SD of the value predicted by using the clinical models. Only 69% of the differences based on the unrestricted administrative model and 77% of the differences based on the restricted administrative models met this criterion.

Table 4 compares selected ICD-9-CM codes with the comorbid conditions and laboratory values documented in patients' medical records at hospital admission. Of the 13 comorbid conditions and complications examined, 6 were documented as present at admission less than half the time. For example, of the 831 codes for cardiac arrest during hospitalization, 70.4% were recorded after the day of admission. Similarly, only 12.4% of patients with ICD-9-CM codes for shock had systolic pressures lower than 90 mm Hg at admission.

To determine whether hospital-acquired complications might explain large differences in mortality predictions between the unrestricted administrative models and the clinical models, the incidence of complications (ICD-9-CM code without documentation supporting the condition's presence at hospital admission) was examined whenever the same patient's risk for dying was found to differ by three or more deciles (Table 5). For all conditions, more hospital-acquired complications were seen when the unrestricted administrative model's estimate of risk exceeded the clinical model's estimate than when the reverse was true. For example, of the patients who had acute myocardial infarction and whose predicted probability of dying was three or more deciles greater when the unrestricted administrative model was used than when the clinical model was used, more than 23% had ICD-9-CM codes for shock or hypotension but no clinical evidence of these conditions at hospital admission. However, when the probability of dying predicted by the clinical model was at least three deciles greater than the probability of dying predicted by the unrestricted administrative model, fewer than 2.0% of patients with acute myocardial infarction had ICD-9-CM codes for shock or hypotension without supporting documentation at hospital admission

Discussion

To eliminate the effect of differences in analytical methods among models, the same researchers developed and evaluated all predicted probabilities of hospital mortality rates in an identical manner. Therefore, the only factor that influenced the predictive power of corresponding risk-adjustment models was the set of data from which each Equation was developed. The clinical models (which used restricted administrative plus laboratory and clinical data) served as criterion standards. The predictive accuracy of these models, measured by using the areas under ROC curves, almost always exceeded that of the other models and compared favorably with predictions published for similar patients who were evaluated using current commercial “severity” systems, regardless of the source of data [13, 21-23]. The substantial predictive power of the clinical models used in our study supports the use of these models as a criterion standard for the predictive capacity of the current generation of “severity” systems that adjust hospital mortality rates for risk among patients with acute myocardial infarction, cerebrovascular accidents, congestive heart failure, or pneumonia. However, validation of these models, ideally in other populations, is needed before their performance is definitively established.

The strikingly worse mortality predictions provided by the restricted administrative model as compared with the clinical model are consistent with those seen in previous reports [10, 14]. Comparisons of patient-level and hospital-level mortality predictions derived by using different sources of data further confirm the value of supplementing restricted administrative data with clinical or laboratory data. The tendency of the restricted administrative model to underestimate the risk of the sickest patients for dying (as identified using clinical data) could undervalue the performance of hospitals that treat high-risk patients while unduly crediting hospitals that have healthier inpatient populations.

The finding that the unrestricted administrative model predicted mortality as accurately as did the clinical model is similar to the finding of Iezzoni and colleagues [13]. However, our study confirms that much of the predictive capability of the unrestricted administrative model is due to the inclusion of numerous hospital-acquired complications that commonly precede death.

Despite the obvious inadequacies of solely administrative models, many errors in patient-level predictions may offset one another when individual patient risks and outcomes are aggregated and standardized mortality ratios of hospitals are compared [23]. For the four medical conditions studied, however, differences in risk-adjusted hospital mortality rates of more than 1 SD were not uncommon when predictions based on clinical data were compared with predictions based on administrative data alone. These differences were due to incorrect individual predictions obtained by using restricted and unrestricted administrative models and to the systematic tendency of restricted administrative models to underestimate patient risk in hospitals serving higher-risk patients. Risk-adjusted hospital mortality rates based on the laboratory models rarely differed by more than 20% from rates based on the clinical models.

Thus, adding laboratory data to restricted administrative information from patients' hospital billing records resulted in models that, at both the patient level and the hospital level, predicted death almost as well as did models that require abstracting data directly from written medical records. Therefore, accurate patient-based predictions of death for risk adjustment can be obtained by merging these two data sets. This avoids the considerable costs of reviewing and abstracting individual medical records.

Several methodologic limitations of our study should be acknowledged. First, administrative and laboratory data were obtained by abstracting medical records. Although additional studies are needed to show similar results with electronically derived data, the predictive power of our administrative models was similar to that of models developed from computerized hospital billing data. Moreover, previous studies have shown improvements in predictions of length of hospital stay when electronically derived laboratory data were combined with electronically derived administrative data [24].

Second, although our models evaluated four common and previously well-studied conditions, further studies are required to determine whether our findings can be generalized to other conditions. For instance, accurate predictions of hospital mortality rates for patients having surgery may require data that are not readily available electronically (for example, cardiac ejection fractions for patients having coronary artery bypass surgery [14] and anatomical characteristics of lesions being dilated for patients having coronary angioplasty [25]).

Third, although our findings relate to a diverse group of hospitals, their applicability to hospitals in other geographic regions should be established. It should also be recognized that imputation of missing values is a complex statistical problem. The method used to impute missing values can affect both the estimates of coefficients and SEs in a predictive model and, therefore, the results [26].

Our findings have important implications for public policy. They support and supplement studies showing that hospital mortality rates based on administrative data alone may be biased and misleading [13, 27], even when c-statistics show excellent predictive accuracy. Therefore, administrative data alone should never be used for a definitive study of comparative clinical performance. If more accurate data are unavailable, administrative data alone may be used for initial screening or to guide future studies, but findings must always be considered tentative, not conclusive. If laboratory data are accessible, it may no longer be cost-effective for hospitals to insist that public monitoring of hospital mortality rates be based on administrative data alone.

For hospitals internally studying comparative risk-adjusted mortality rates to guide quality improvement, predictions from administrative data alone may be similarly difficult to justify. Attempts to compare the effectiveness of different practice patterns or to determine the quality of care provided by individual physicians often require analyses of relatively small groups of patients who may be far more homogeneous than the entire population of patients hospitalized at a single facility for a specific condition. In these instances, errors introduced by inaccuracies in risk-adjustment models based on administrative data alone can be large. When real inadequacies in clinical care are not accurately identified by administrative data alone, costs of misdirected quality improvement initiatives can far exceed the costs of merging administrative and laboratory data to permit more accurate predictions of patient risk.

Our study has shown that the risk for dying during hospitalization for any of four important medical conditions can be predicted by using billing and routinely collected laboratory data without acquiring clinical information from patients' medical records. The development of good statistical models and the imputation of missing values, however, require considerable clinical and statistical expertise. Nonetheless, our results suggest that accurate comparisons of risk-adjusted hospital mortality rates can be made relatively inexpensively, both to monitor hospital performance and to improve the quality of medical care.

Appendix

A. ICD-9-CM Codes Used To Identify Medical Conditions

Acute myocardial infarctions (6088 patients) were identified by principal diagnostic codes 410.xx, where x represents all integers from 0 through 9. Cerebrovascular accidents (9061 patients) were identified by principal diagnostic codes 431.xx, 432.9x, 433.x1, 434.x1, and 436.xx. Congestive heart failure (18 864 patients) was identified by principal diagnostic codes 398.91, 402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, 404.93, and 428.xx. Pneumonia (12 756 patients) was identified by principal diagnostic codes 480.xx, 481.xx, 482.xx, 483.xx, 485.xx, 486.xx, 487.0x, and 507.0x.

B. Data Elements Used in Mortality Risk-Adjustment Models, Classified according to Type of Data

Administrative Data

Age, sex, admission source (for example, emergency department or nursing home), descriptive information contained in principal diagnostic ICD-9-CM codes (for example, location of an acute myocardial infarction, causative agent in pneumonia, or hemorrhagic versus ischemic cerebrovascular accident), and chronic diseases identified as secondary diagnoses by ICD-9-CM codes (for example, cancer, chronic obstructive lung disease, chronic renal failure, diabetes, hypertension, or malnutrition).

Laboratory Data

(These data consist of the worst values obtained during the first 2 days of hospitalization. Abstractors copied both the highest and the lowest values of each test result during the first 2 days, except for tests for which either the lowest or highest value is known to be the worse. In this case, only the appropriate value [for example, the highest blood urea nitrogen level] was requested on the abstraction form.)

Blood chemistry variables (for example, levels of albumin, alkaline phosphatase, bilirubin, blood urea nitrogen [set to 100 mg/dL for patients receiving long-term dialysis], calcium, creatine phosphokinase, glucose, lactate dehydrogenase, potassium, aspartate aminotransferase, or sodium), blood gas variables (for example, PO₂, PCO₂, or pH), and hematologic variables (for example, hematocrit, platelet count, prothrombin time [set to “missing” for patients receiving anticoagulant therapy], or leukocyte count).

Clinical Data

(These data consist of the first values obtained during the first 2 days of hospitalization.)

Cardiac or respiratory arrest immediately before admission, chest radiographic findings (for example, cardiomegaly or pulmonary edema), electrocardiographic findings (for example, atrial fibrillation or bundle-branch block), medical devices present on admission (for example, feeding tube or urinary catheter), motor function (for example, paralysis or paresis), neurologic status (for example, coma or disorientation), and vital signs (for example, body temperature, pulse rate, respiratory rate, or systolic blood pressure).

Dr. Rosenthal: Section of General Internal Medicine, Cleveland Veterans Affairs Medical Center 111G (W), 10701 East Boulevard, Cleveland, OH 44106.