The problems highlighted by Iezzoni and colleagues might well be worse than the study indicates. First, these authors studied patients with acute myocardial infarction, an acute illness that has short-term mortality and requires hospitalization. For diseases with mortality rates far lower than the 13% seen in this study, statistical models generally perform less well and demand more data. Almost all patients with acute myocardial infarction are admitted to nearby hospitals, where their outcomes can be readily audited using conventional hospital data systems. In contrast, in patients having procedures such as coronary artery bypass surgery, the underlying disease is treated alternatively without surgery in the outpatient setting. The prudent investigator must consider selection bias, that is, the effects of the decision to choose surgery on the outcome of patients who elect to have surgery [2, 3]. If the outcome is changed from inpatient mortality to 1-year survival or nonfatal adverse events or length of hospitalization, the comparative performances of different systems undoubtedly will change [4].

Second, Iezzoni and colleagues [1] stop short of addressing another problem: a gap between user expectations and reality. The expectation is that a simple analysis of the results of severity-of-illness systems will produce valid and reliable comparisons of provider performance. In reality, however, the statistical issues in these comparisons are difficult and not well understood [5]. A gap exists between the introduction of new statistical methods and their use in clinical research [6], and an even greater lag separates the publication of the statistical methods needed to implement these systems and the proper use of these systems by employers, insurers, managed care organizations, and governments [7]. Nonetheless, these systems are thought by some to be safe and reliable.

The danger and effectiveness of these severity-adjustment tools depend on the manner in which they are used. When hospitals and physicians are compared using performance "report cards," a risk of harm arises if these systems are applied blindly or myopically. Patient referral patterns might result in the clustering of especially sick or unusually robust patients in particular hospitals or practices. Severity algorithms might operate unevenly, resulting in systematic biases in comparisons across providers. At the same time, the number of patients treated by a single hospital or physician in a typical review period frequently falls well below the hundreds needed to detect meaningful differences in rates of mortality or complications. Nonrandom measurement bias in estimations of severity of illness, together with random variation in outcome in small samples, can lead to false conclusions about performance, the dropping of physicians by managed care plans, and the unnecessary termination of long-lasting and high-quality physician-patient relationships.

Severity-of-illness algorithms would pose even greater risks if individual decisions about patient care were driven by assessments made solely on the basis of the algorithms. This hazard is easily avoided by the treating physician, who has other sources of information with which to temper the computer's assessment. But managed care systems now include utilization reviewers, who make off-site decisions about admissions, treatment plans, and discharges. A flawed severity algorithm in the wrong hands could inflict direct harm on the patient who cannot afford uninsured treatment outside the managed care plan.

New hopes and new dangers are on the horizon. One hope lies in the automated link between the clinical information system of the integrated health care network and the severity-of-illness system [8]. These links could reduce at least one source of error: the inconsistent coding of diagnoses by patient and hospital. One danger arises from increased access to, and thus an increased potential for misuse of, on-line severity-adjustment systems. Information automation will not resolve the ethical and statistical issues involved in using these systems to decide who should treat whom for what and for how long.

Despite the growing number of articles and editorials on the policies and limitations of these statistical systems in comparing provider performance [9-15], report cards that disregard published recommendations and cautions continue to appear. An analysis of interests might explain why. Those who pay for care do not need scientific accuracy if their overriding goal is to reduce the costs of medical care. Even crude report cards create an opportunity to demand price concessions, initially from the providers who do not receive the highest ratings and subsequently from those who do. Differences in the performance of severity-of-illness systems, together with random variation, almost ensure that next year's mortality rankings will differ from today's. Although the U.S. government has supported the research of Iezzoni and colleagues and many related projects, government policies on the regulation and monitoring of these widely used computer programs are only just evolving [16]. Under employer-funded managed care, the rights of employees and patients to question the rationale for selecting or dropping participating physicians and hospitals are severely limited [17, 18].

The health care community has both an interest in improving and an opportunity to improve the integrity of both the data and the computer programs used to measure severity of illness. As a forum for scientific review of these clinical tools, Annals of Internal Medicine and other peer-reviewed journals can help the medical community to become fluent in, rather than merely conversant with, the data and methods of performance assessment. The quality of patient care might be at its highest when the medical community thoroughly understands these severity-of-illness systems and, thus, knows when to act on and when to disregard the results these systems produce. The medical journals also bear the responsibility for objective reporting to all interest groups. The typical purchaser, who is probably a legislator, an insurance executive, or an employee-benefits manager, would find it difficult to competently assess the merits of these severity-of-illness systems, even if he or she could obtain complete information on their design and performance. All persons need simple, objective information about these systems—just as they need information on drugs and treatments—to protect their interests as informed patients. The medical journals must be to these systems what Consumer Reports is to automobiles. The report cards need report cards.

Managed care is here to stay, as are systems to assess performance and the public disclosure of performance scores. As new systems appear and old ones mature, the standard of review should always relate to the effect of systems on access to and quality of medical care. The biomedical research community, whose clear goal should be to protect patients, should continue to assess these systems as carefully as it would new drugs or treatments.

Dr. Hamory: Office of Clinical Affairs, Pennsylvania State University College of Medicine, P.O. Box 850, Hershey, PA 17033-0850.