In general, physicians and hospital administrators react defensively to the concept that medical outcomes should be measured, and they worry about the consequences of these data being made public [4]. In particular, they worry about being discriminated against. These worries are not unfounded, because perceptions of quality appear to play a role in choice of hospital [5]; hospitals with high mortality rates can expect not only negative publicity but also fewer patients in the future. But is a hospital designated as a high-mortality outlier because of deficiencies in quality or because of the characteristics of the patient populations it serves? Although most physicians are unfamiliar with methods of risk adjustment and are skeptical about their accuracy, it is increasingly important that clinicians understand these methods.

Outcomes of care in hospitals are a function of patient characteristics, quality of care, and chance. When hospitals differ, it is important to know how much of the variation results from differences in severity of illness, how much from quality of care, and how much from random events. A hospital with average quality of care may have higher-than-average mortality rates because its patients are sicker or more disadvantaged than average. A hospital may have higher-than-predicted mortality rates if these and other factors are not controlled for in the predictive model. Methods for the adjustment of risk project the outcomes that could be expected if the same patients were admitted to each hospital. An immense array of variables are available to describe a patient's condition. The aim of risk adjustment is to find a parsimonious representation of those patient characteristics that have a strong relation to mortality and that could, without proper attention, confound the results of a particular analysis. This is done by fitting statistical models to approximate the relation of patient characteristics to observed outcomes.

Localio and colleagues [3] show that an inadequate strategy for risk adjustment can lead consumers of information on quality of care to the wrong conclusions. Selected in part on the basis of a consultant's findings, several hospitals participated in a company's managed care network. It is likely that one or more of the hospitals not selected may have suffered financial hardship.

The consultant's methods were flawed in two major respects. First, the consultants compared the death rates for each hospital with the overall average without adequately considering the statistical significance of these differences. Second, their model for risk adjustment did not include age, an obvious prognostic factor and one that differed among institutions. When Localio and colleagues [3] repeated the analysis with more conventional statistical methods, the hospital that had had the worst performance no longer differed significantly from the other hospitals. When age was considered, this hospital looked even better, and the rank order of hospitals changed. This is consistent with studies showing that adjustments for age and other factors can alter conclusions about quality of care [6, 7].

A more subtle point concerns the importance of random events. Not all outcomes are within the control of the physician, and not all outcomes are predictable. Consistent with previous findings [8], simulations done by Localio and colleagues [3] showed that comparisons of multiple hospitals resulted in random variation being attributed to hospital quality. This misattribution was reduced by selecting more stringent P values when mortality was more common and when larger numbers of patients were evaluated.

In analyses by the consultants, and in re-analyses by Localio and colleagues, the MedisGroups admission severity groups score was used to assess severity of illness [9]. One strength of the MedisGroups system is that it uses data from the medical record and thus incorporates more clinical detail than other systems. There are, however, reasons why it is less than ideal for some comparisons of hospital quality. First, to accurately capture a patient's risk for death at hospital admission, predictions should be based on initial presentation. The MedisGroups score is based on findings from the first 48 hours of hospitalization and may therefore already reflect suboptimal care. For acute pulmonary problems, for example, interventions within the first hours are important; such problems may account for a large proportion of preventable deaths. Second, a model to predict mortality should need only data that are collected in the usual care of patients. If, as in the MedisGroups system, weight is given to special tests, then a hospital's apparent performance will be enhanced by using more tests [10]. These methodologic shortcomings have been shown in previous studies [11, 12]. Finally, to facilitate acceptance, the components of the system should be open for inspection and testing. MedisGroups, however, is a proprietary "black box" system.

Although prediction was not an aim of the study by Localio and coworkers, we do not know how well their models predicted. Predictive models should be well calibrated, that is, they should predict accurately at all severity levels to avoid creating disincentives to caring for sicker patients. Doubtless, Localio and colleagues' risk model omitted other important variables. Data from other studies [13] suggest that variables in addition to age, such as pleuritic chest pain, may be important predictors of death in patients with pneumonia. It is also important that the at-risk population be defined appropriately. For example, hospitals that provide substantial terminal care should not be penalized for higher rates of death. Mortality, of course, is not the sole measure of outcome. For some conditions, particularly those in which death is infrequent, functional status and quality of life may be more relevant. Outcome comparisons must eventually include comparisons of effects on quality of life, and, in turn, risk-adjustment methods must be developed for these outcomes.

Data on hospital outcomes may be used to improve quality or to compare hospitals for consumer choice. Data comparing hospitals may be prepared for release to the public or to be used confidentially by buyers to select providers. The level of accountability required of data analysts differs for different applications [14]. There is little controversy about internal use of outcome data for quality improvement; results used in this way are unlikely to harm providers and low thresholds may be used to identify potential problems. Greater caution is needed when generating data for comparing hospitals. Higher thresholds should be set for identifying outliers to spare hospitals from being tarred inappropriately. The general public is risk-averse and may react more emotionally than do researchers to statistically "insignificant" findings. In the analysis by Localio and colleagues [3], the odds of death at the original high-outlier hospital were still 1.8 times higher than expected, although the P value was 0.20. Patients are likely to avoid a hospital with poorer-than-expected outcomes even if there is a 1 in 5 chance that this conclusion is incorrect. It will be important to develop presentation strategies, such as combining multiple years of data, that minimize the effect of random variation on performance figures.

Large employers and other institutional buyers should be expected to base their decisions on data that accurately reflect hospital quality; this is important to their employees as well as to hospitals. Although no published data support this point, there are suggestions that statistical malpractice is not uncommon. Who should watch this increasingly diverse group of doctor-watchers? Risk-adjustment methods used by the Health Care Financing Administration or the State of Pennsylvania come under substantial scrutiny, but comparisons made on behalf of employers, insurers, and other private organizations are generally made less openly, and the analyses are often viewed as proprietary and confidential. One solution would be to formulate and disseminate guidelines for acceptable risk-adjustment methods. These guidelines could target both analysts and consumers of the data and might include standard statistical methods, acceptable severity-of-illness systems, the minimum number of cases needed, and informative reporting formats. Lawsuits may be brought by unfairly identified hospitals against consultants that provide flawed "buy right" data. Alternatively, regulation or certification of these companies may be needed.

The demand for accountability by health care providers must be balanced by responsible evaluation to assure valid conclusions. Drawing conclusions from aggregated outcomes data without properly accounting for bias, confounding, and statistical power can lead to erroneous inferences and policies. As public and private sector initiatives to assess quality to guide consumer choice expand, measurement and analytic practice should keep pace with scientific standards. Effective strategies to develop, apply, and disseminate performance data will be vital to improving decision making and quality of care.