A less frequently discussed but equally vital issue in the comparison of patient outcomes is the appropriateness of the statistical methods used for analyzing data and interpreting results. Raw data are clearly insufficient for comparing hospitals or physicians; some statistical methods must be invoked. Ideally, using a set of techniques called standardization [18], the comparison of different samples of patients in varied clinical settings should be feasible.

We examine one attempt to compare hospitals based on their observed mortality; this attempt was made in 1992 by consultants to a managed care program for a large corporation. The corporation sought to determine which of the hospitals serving the corporation's employees in central Pennsylvania delivered better quality of care as reflected in part by fewer in-hospital deaths in 1989 and 1990. Partly on the basis of the methods reported here, the corporation selected 10 of these hospitals to be eligible for its managed care network.

Focusing on adult pneumonia, which was one of the diagnostic groups used by the consultants, we describe the consultants' methods and re-create their results for 1989 and 1990. Then, we reassess those results using more appropriate methods, and finally, we validate results using newly acquired 1991 data. (A glossary of statistical terms is given in Appendix 1.)

Methods

Top Methods Results Discussion Author & Article Info References

Choice of Diagnosis-related Groups

The study sample included data reported to the Pennsylvania Health Care Cost Containment Council on all inpatient hospitalizations for adult pneumonia (diagnosis-related groups 089-090) in 1989 and 1990, and, later, in 1991, for 22 hospitals in central Pennsylvania. Because we were using the consultants' methods, we excluded from analysis all patients older than 65 years of age, for whom Medicare would likely be the primary insurer. In addition, we excluded four patients with the acquired immunodeficiency syndrome and seven awaiting organ transplantation because only one hospital provided these specialized kinds of care. Examination of the 1991 data did not begin until after completion of the re-analysis for the years 1989 and 1990.

The public-use data sets include coded discharge abstracts and the MedisGroups (Mediqual Systems, Westborough, Massachusetts) admission severity group, an indicator of severity of illness [19]. Admission severity group attempts to distill many key clinical findings at patient admission into a single, five-level severity score. In the years 1989-1991, admission severity group was not diagnosis specific; a patient received a given admission severity group score regardless of the cause of his or her illness. All hospitals in Pennsylvania must subscribe to the MedisGroups system, calculate admission severity groups retrospectively from medical records, and report these data and computerized discharge abstracts to the Cost Containment Council.

The Managed Care Consultants' Analysis

The consultants relied only on admission severity group to adjust for both severity of acute illness and comorbid conditions. Patient age, which is not an element in admission severity group, was not used. The consultants' statistical technique is best described as a stratified, standard, normalized analysis. First, they calculated death rates for each admission severity group within each institution. Next, they computed the unweighted mean and standard deviations for admission severity group death rates for all hospitals. They then computed a standard, normalized [20] difference between the hospital and the average death rate for each level of admission severity group by dividing the difference by the standard deviation of death rates. Finally, they combined these normalized differences across diagnosis-related group levels within each hospital by a weighted average. Weights depended on the fraction of the hospital's admissions in each admission severity group. The consultants did not calculate standard errors or P values for these weighted averages of normalized differences; they displayed their results along a horizontal line, indicating the location of the average normalized difference and the relative position of a given hospital along the line. Further details of our implementation of the consultants' methods are given in Appendix 2.

Methods of Re-analysis

Our re-analysis of the data on adult pneumonia from the same 22 hospitals sought to confirm or refute the consultants' results with conventional methods for data with binary outcomes, in this case, death or survival at discharge.

We implemented model-based standardization [21] to compare hospital mortality adjusted for multiple patient risk factors. First, we verified the association between death and the patient's admission severity group by logistic regression as implemented in SAS, Proc Logistic (SAS Institute, Cary, North Carolina). Next, the hospital with the worst outcome in 1989-1990 (hospital 1) as calculated by the managed care consultants' method Table 1, was compared with the others by fitting two "nested" regression models: one with only admission severity group as a covariate, the other with admission severity group plus an indicator variable to detect any difference between hospital 1 and all other area facilities. The likelihood ratio statistic for the differences between the two models showed whether the hospital with the worst outcome, according to the consultants' analysis, was truly different from the other hospitals.

Also relevant to the outcome of many acute diseases is patient age, a factor readily available in the public-use data set but not used by the consultants. Based on a preliminary calculation showing that mortality increased with patient age, patients were grouped into three age ranges: 18 to 39 years, 40 to 54 years, and 55 to 64 years. For age to affect interhospital comparisons of pneumonia mortality, two conditions must be met: The distribution of ages must differ across hospitals and age must be associated with death after controlling for admission severity group. We tested the first condition by cross-classifying the three age categories and the 22 hospitals and calculating the Mantel-Haenszel chi-square statistic for differences in mean age across hospitals. We tested the second condition by using the likelihood ratio statistic for the difference in two logistic regression models: one with both admission severity group and age as factors and the other with only admission severity group as a factor.

The last step in testing whether hospital 1 differed significantly from the others again involved the comparison of two nested models. For this comparison, the two models included both age and admission severity group as standardization factors, and one also included an indicator variable for hospital 1. The ratio of the odds of mortality and a P value were computed as previously described.

After determining whether hospital 1 had an adjusted mortality higher than that of the other hospitals, we investigated whether another hospital among the 22 might have had significantly higher mortality than expected. Logistic regression also allows for the calculation of predicted probabilities of death for each patient with a given age and admission severity group level. By summing the number of deaths and this predicted probability of death over all patients with pneumonia within each hospital, we computed at the hospital level both an observed and a predicted (or expected) number of deaths.

Hospital 4 showed the largest positive difference between observed and predicted numbers of deaths and thus became the test hospital for further analysis of differences in mortality. As before, we fit two nested regression models, one with admission severity group and age as covariates and the other with an additional single indicator variable for hospital 4, to test whether patient mortality was significantly higher there than at other facilities.

Multiple Comparisons

Contrasts among hospital death rates in the consultants' study, as in typical outcomes analyses, were determined by the data rather than planned in advance. Each hospital was compared in turn to determine whether it differed from all others. In this situation, a critical P value of 0.05 used repeatedly for each hospital comparison did not function to limit type I error, which was the probability of falsely classifying a hospital as a mortality outlier. Diehr and colleagues [22, 23] have commented on the issue of multiple comparisons, or multiple statistical tests, in a related context. The Bonferroni method [24], one of several available techniques [25], was used to adjust the critical P value to control type I error for multiple comparisons of hospitals.

Computer Simulations

Computer simulations were done to determine the overall type I error rates caused by using various critical P values for significance tests. Multiple comparison procedures can affect type II error rates while they control type I error. In this context, type II error is the probability of failing to identify a true mortality outlier [26]. Simulations that used the actual pneumonia data from the 22 hospitals and that applied the logistic regression model described previously produced estimates of both types of error under different conditions.

Simulation of type I error involved generating deaths randomly by computer for each hospital. Each patient with the same age and the same level of admission severity group was assumed to have the same probability of death, as predicted from the logistic regression model. The computer simulated the chance or random death of a patient with this predicted probability of death. At the hospital level, the sum of the individual random deaths became the simulated number of random hospital deaths; the sum of the patient-level predicted probabilities of death became the expected number of deaths. Each simulation thus produced a difference between a chance observed death rate and an expected death rate, and this difference was transformed into a standardized difference in the two rates for each hospital [27]. This was not, however, the same standardized difference as computed by the consultants: Our differences were computed at the hospital level rather than at the much smaller admission severity group levels within each hospital. The standardized difference, or z statistic, was then compared with two critical values for type I error: the typical critical P value of 0.05 (corresponding to a z statistic 1.96) and a smaller critical P value of 0.002 (corresponding to a z statistic 3.08) adjusted for multiple comparisons by the Bonferroni method. The latter critical P value is simply 0.05 divided by 22, the number of hospitals being compared simultaneously. A hospital with a z statistic larger than that corresponding to the chosen critical P value in any simulation was a false-positive high outlier. The number of false-positive outliers in 1000 simulations provided an estimate of overall type I error.

The estimation of type II error was done in a similar way. Hospital 4 again became the test case. Each simulation generated a random number of deaths for each hospital. This time, however, we altered the true probability of death for each patient at hospital 4 to be greater than average. We estimated type II error as the number of simulations in which hospital 4 was missed as a high outlier. The 1000 simulations also tested whether a Bonferroni-adjusted critical P value of 0.002 would result in the loss of statistical power to detect a true mortality outlier (increased type II error).

Finally, we simulated the effect of larger and smaller overall death rates for pneumonia, and of larger numbers of patients with pneumonia at all 22 hospitals, on the probability of type I and type II errors. Each simulation was done 1000 times.

Validation with 1991 Data

After completing all the analyses described above, we analyzed the 1991 pneumonia data for the 22 hospitals. We computed crude death rates for each hospital Table 1 to compare them with the ranking of hospitals done by the managed care consultants for 1989-1990.

Results

Top Methods Results Discussion Author & Article Info References

The overall mortality rate for adult patients with pneumonia who were less than 65 years of age for the 22 central Pennsylvania hospitals was 3.48% (80 of 2297) for 1989-1990. Raw death rates by hospital ranged from 0% to 8.0% (Table 1). Numbers of beds per hospital ranged from 79 to 553, and numbers of hospitalizations for pneumonia at each hospital varied more than sevenfold, from 28 to 203 during this 2-year period. Mortality increased sharply with increases in both admission severity group and age. When the effect of admission severity was held constant, mortality continued to increase significantly with age (likelihood ratio test, P < 0.001) (Table 2).

Managed Care Analysis

The consultants' methods produced hospital-specific standard normalized death rates ranging from –0.69 to 2.35. (Table 1). The consultants considered all eight hospitals with standardized death rates greater than the average (greater than 0.0) to have poorer-than-average performance for this diagnosis. Hospital 1 fared the worst with a raw death rate of 5.32% (5 of 94); its normalized rate (2.35) was more than two standard deviations above the average and was substantially higher than that of the second worst institution (0.90). The hospital with the highest raw death rate did not rank as the worst because of adjustment for patients' severity of illness (admission severity group scores).

Re-analysis

When conventional statistical methods for assessing categorical data were applied, mortality at hospital 1 did not differ significantly from that of the 21 other area hospitals (likelihood ratio test, P = 0.13), even when severity adjustment was made only by admission severity group. Data available to the managed care consultants showed clearly that the 22 central Pennsylvania hospitals had significantly different distributions of patients with pneumonia by age (Mantel-Haenszel chi-square, P < 0.001); hospital 1 had the largest percentage (53.2%) of patients in the highest age group (55 to 64 years). Age was also associated with mortality even after adjustment was made for admission severity group; this omitted factor was therefore a confounding variable [28]. After adjusting for both age and admission severity group, the odds of death were still higher at hospital 1 than at other hospitals (odds ratio, 1.8), but the odds were lower than the comparable Figure adjusted only for admission severity group (odds ratio, 2.3). The P value for the difference between hospital 1 and the other hospitals increased from 0.13 to 0.23 using the likelihood ratio test. The re-analysis, therefore, found no difference in mortality between hospital 1 and other area hospitals.

Our further analysis of hospital mortality, standardized by both admission severity group and age, produced a different ordering of adjusted hospital death rates (Figure 1). Hospital 1 now ranked fourth highest; hospital 4 ranked highest because it showed the largest positive difference between observed and expected death rate (3.7 percentage points). The odds ratio for mortality at hospital 4 compared with all other hospitals was 2.60 (likelihood ratio statistic, P = 0.012). Had we hypothesized before the analysis that this hospital alone would be compared with all others, this result could have been compared with conventional levels of statistical significance, such as the commonly cited critical P value of 0.05 [29]. However, the comparison was post hoc; it was suggested by the data after an analysis that involved multiple comparisons. Therefore, we adopted a smaller critical P value (P = 0.002).

View larger version (20K): [in this window] [in a new window]   Figure 1. Adjusted hospital mortality. Ranking and statistical significance. Normalized differences (z) between observed and expected mortality for 22 central Pennsylvania hospitals using the managed care method and our re-analysis. z = ±3.08 corresponds to a critical P value of 0.002 (adjusted for multiple comparisons), and z = ±1.96 corresponds to a critical P value of 0.05 (no adjustment). Several hospitals' ranks change markedly with re-analysis. No hospital remains an "outlier" when a critical P value of 0.002 (z = ±3.08) is applied.

Simulations

Computer simulations offered insight into the critical P value (or adjusted z statistic) that should apply to the classification of individual hospital outliers in order to maintain a type I error of approximately 5% for all hospital comparisons. With a critical P value of 0.05, 59.4% of the simulations erroneously classified at least 1 of 22 hospitals as a "high-mortality outlier" (type I error). A conventional critical P value thus produced many false findings when random variation was the only cause of differences in mortality.

Other simulations showed the interrelation of the overall level of mortality, the number of patients in the sample, the size of the true difference between observed and expected mortality, the critical P value, and type I and type II errors. A simulation with a critical P value of 0.05 but with a lower overall mortality for all patients with pneumonia (one fourth the predicted death rate) resulted in a higher chance of falsely classifying at least one hospital as a high-mortality outlier (67.9%); a higher death rate (double the predicted rate) reduced that chance (49.9%). Doubling the number of patients in each hospital while holding mortality constant lowered the chances of false outliers from 59.4% to 51.5%.

The Bonferroni adjustment, made to account for implicit, multiple comparisons among 22 hospitals, reduced the critical P value to 0.002. Computer simulations with this P value showed a decrease in the overall type I error (the chance of a false finding of at least one high outlier) to 7.9% (Table 3). As expected, this adjustment to the critical P value did increase the type II error (the probability of missing a true high-mortality outlier). For example, the type II error for hospital 4 was 0.626 (62.6%) when the true mortality for each patient in that hospital was simulated at twice the expected rate. The type II error decreased to 0.244 (24.4%) when true mortality was simulated at 2.5 times the expected rate, and it decreased to 0.004 (0.4%) with a simulated doubling of the number of patients in each institution.

The simulations thus showed the high probability of type I error with mortality data of this type. Statistical power improves (type II error decreases) when a hospital's true death rate is several times higher than expected or when the true death rate is elevated and the number of patients in the sample is larger. When a critical P value of 0.002 was used, hospital 4 did not attain the requisite level of statistical significance for an outlier designation for 1989-1990.

Confirmation with 1991 Data

The follow-up data from 1991 corroborated our assessment that the managed care consultants' methods falsely classified hospital 1 as having the worst mortality record (type I error). In 1991, this hospital had no deaths among 47 patients. The rate of admission was about the same as in the previous 2 years, and the 1991 admissions occurred before the conclusion of the managed care study: The consultants' report could not have affected clinical care for 1991. Two other hospitals that had higher-than-average mortality in 1989-1990 also had no deaths in 1991. Hospital 4, which had the highest but still did not have significant adjusted mortality in our re-analysis, had average mortality in 1991.

Discussion

Top Methods Results Discussion Author & Article Info References

Other investigators have cautioned against using mortality data to compare hospital performance [30]. Studies have focused on data or scoring systems, largely untested [31], for determining illness severity at admission as distinguished from severity because of treatment. We focus on the effect of inappropriate statistical methods, a topic that has received little attention in the medical literature, perhaps because physicians are less attuned to statistical than to clinical issues. Using actual data and relying on basic statistical concepts of small sample sizes, low death rates, and multiple comparisons, we show how a misapplied statistical analysis of public data can generate specious results and incorrect conclusions.

The example of adult pneumonia is particularly instructive. Most hospitals admit patients for pneumonia; inpatient deaths for patients under the age of 65 are relatively rare. Often, there are no deaths in subgroups of patients defined by the cross-classification of diagnosis, age, and severity [32]. In these cases of rare outcomes and limited sample sizes, commonly used statistical methods that might be satisfactory for large samples can work poorly [33].

We used logistic regression to standardize interhospital differences in patients and to test for differences between one hospital and all others. Contingency Table analysis, Cochran-Mantel-Haenszel methods [34], and exact methods [35], which are especially appropriate when deaths are few, confirmed our findings. Grouping patients might provide samples large enough to justify the use of simplistic statistical methods but at the cost of introducing bias and defeating the goal of standardization if dissimilar patients are grouped together. For example, the consultants grouped patients regardless of age, but we identified age as a textbook example of confounding; inclusion of age improved the relative performance of hospital 1, which cared for an older patient population. It is likely that additional factors might explain some of the remaining variation in mortality; for example, the case-fatality rates for pneumonia vary according to causative organisms but such differences [36] might not be reliably ascertained from an administrative database. The omission of relevant clinical factors can lead to biased results. It can also make results more uncertain by increasing variance [37].

Our computer simulations show the danger of implicit, multiple comparisons in these analyses. The interplay of type I error and the number of comparisons is a fundamental concept in statistical methods [38, 39]. Our simulations show that this danger of false-positive findings can be high and will increase with smaller numbers of patients and with lower average mortality rates. Yet, use of 0.05 as the critical P value is common among investigators comparing multiple hospitals simultaneously [4, 40].

Methods other than the one we cite are available to control for multiple comparisons [41]. Computer simulations can also serve this purpose by estimating both type I error (false-positive results) and statistical power (the power to uncover true differences) with actual data. These estimates can then suggest appropriate critical P values for testing hospital differences.

The possibility of finding a false hospital outlier could be confirmed by an examination of data from another year. If the outlier hospitals change from year to year, one should suspect the cause to be random variation rather than secular change; this is precisely what we found. Data from 1991 found the "worst" hospital (hospital 1) becoming one of the "best." In all likelihood, nothing actually changed; random variation explains the difference.

In focusing on statistical issues, we do not advocate using the most elaborate or computer-intensive models. Exploratory data analysis [42] might be a helpful initial step, provided that the investigator appreciates the tendency of the untrained eye to perceive regularity or clustering where outcomes are in fact random [43]. These and other methods can be especially appropriate for screening; for example, the Health Care Financing Administration uses complex survival models. Nevertheless, its publication of hospital-specific mortality contains this cautionary note: "The information in this release is not intended as a direct measure of quality of care" (44 [emphasis in original]). More recently, the Administration has declined to publish any mortality results because of the inadequacies of the data [45].

The adverse consequences of the misapplication of statistical methods or the misinterpretation of sound ones, although perhaps only short term at the economic levels [46], might be severe and lasting on the personal level. A managed care program might be able to re-evaluate data and reverse an earlier decision, but not necessarily before physician-patient relationships are disrupted when patients must change hospitals and clinicians to maintain insurance coverage. Competent and efficient providers of care might lose business or even disappear merely because they care for patients whose diseases are more complex than computerized data can describe or because random deaths happened to cluster in the 1 or 2 years for which data are available. Although new, high-quality institutions might eventually grow to replace the old ones, individual clinical careers and patients might suffer permanent harm. Patients' worries about limited choice under managed care will take on added fervor if ad hoc outcome analyses needlessly restrict their choices.

The statistical issues we address are not likely to be resolved soon. Consumer groups want "report cards" to assure professional accountability [47]; managed care programs and government agencies, under pressure to promote competition in price and quality, produce one-time or annual comparisons. Recent reports on the federal health reforms in the United States cite the importance of releasing hospital and physician performance data to the public and to purchasers of health services [48, 49], but relatively scant attention is paid to the statistical methods routinely applied to administrative data. The risk posed by improper analysis to the reputations of skilled providers and to the health of the patients they serve leads us, as it has others [50], to advocate stringent criteria for judging the appropriateness of analytic methods for comparing hospitals and physicians based on patient outcomes.

Appendix 1: Glossary of Statistical Terms

Mantel-Haenszel analysis: A method used to compare two groups (such as two hospitals) on a binary response (such as death/survival) while controlling for risk variables. Easy to do with commercial software. Tolerates small cell counts in the tables formed by the cross-classification of risk factors, hospitals, and outcomes. Gives the same results as model-based standardization. An excellent first step to the analysis of associations among risk factors and outcome.

Model-based standardization: The use of a statistical model to control simultaneously for numerous risk factors that predict whether a patient has a given outcome such as in-hospital death, and thus to standardize patient populations for interhospital comparisons. Equivalent to Mantel-Haenszel methods but more difficult to interpret. Easily handles many risk factors and their interactions and is available in many commercial software packages. Encounters difficulty with small cell sizes, causing program to fail to converge to a solution. Tests of significance must be structured and interpreted with care.

Exact tests: Special statistical methods similar to Mantel-Haenszel procedures but they do not rely on assumptions of large samples for the computation of P values or confidence intervals. Especially useful when samples are small or zero and when outcome rates are low. Increasingly available in specialized commercial software.

Indicator variable: A variable in a regression model that defines two groups (of hospitals, for example). The resulting estimate of this variable from the regression indicates the size and significance of the difference (for example, in mortality) between the two groups.

Nested models: A pair of statistical models, such as those used for standardization, in which the smaller model contains a subset of factors found in the larger model. Used for testing whether the additional variable or variables in the larger model are significant (see likelihood ratio test).

Likelihood ratio test: A statistical test of the difference between two nested statistical models that indicates whether the variables in the larger model and not in the smaller model are significant.

Type I error: The probability that a statistical test or series of tests will find a significant difference or a significant comparison when in truth no difference exists (see multiple comparison procedures).

Type II error: The probability that a statistical test will fail to detect a true difference. Related to sample size and statistical power.

Statistical power: The power of an analysis to detect a true difference with a given sample size and a given type I error.

Multiple comparison procedures: A set of techniques that allows the investigator to make repeated or multiple comparisons among subgroups (such as hospitals) while controlling the level of overall type I error in the analysis.

Simulation: The use of the computer to generate a large number of samples of patients with the characteristics of actual patients and to use these samples to test the performance of statistical tests, such as their power or type I or II errors.

z statistic: Also known as the standard normal deviate, the standard normal variate, the normal variate in standard form, and the standard score. A measurement or proportion that has been rescaled so that its distribution resembles a standard normal distribution with mean = 0 and standard deviation = 1.

Confounding variable: A second risk factor that influences and therefore "confounds" the measure of association between the risk factor of interest and the outcome.

Exploratory data analysis: The use of statistical methods and visual displays of data to identify patterns that warrant further investigation. Possible to do with many commercial software packages.

Appendix 2: Statistical Methods of the Managed Care Program

The managed care consultants calculated the admission severity group (ASG)-specific death rate (r_ha) for each ASG level within each institution. Therefore, r_ha = o_ha/n_ha, where o_ha was the number of outcomes for ASG a and hospital h, and n_ha was the number of discharges for that ASG in that hospital. The consultants then computed the unweighted mean (r_a) and standard deviation (sd_a) for each ASG-specific death rate across all hospitals. For each cell created by the cross-classification of ASG and hospital, they computed a standard normalized difference between the hospital and average death rate: z_ha = (r_ha – r_a)/sd_a.

These ASG-specific standardized rates were combined within each hospital by a weighted average, where the weights for each level of ASG were w_ha = n_ha/n_h, and n_h was the number of cases in hospital h. The weights summed to 1 within hospital. On the basis of this method, the hospital-specific standardized mortality became z_h = Sigma w_ha x z_ha.

A value of z_h greater than zero would indicate a hospital with higher-than-average mortality, and a z_h of 2 would represent a mortality that was two standard deviations above the average.