Conclusions: The Academic Medical Center Consortium hospitals had low rates of inappropriate and uncertain use of CABG surgery, regardless of the criteria used for assessment. Even though surgeons from the Consortium revised the appropriateness ratings extensively, their revisions had a negligible effect on the overall assessment of appropriateness. However, because of potential data collection errors, appropriateness criteria should be used for individual case audits only if supplemented by subsequent physician review.

As the principal resources for innovation and education in health care, academic medical centers have an obligation to advance the quality of medical care. To further this objective, 12 teaching institutions banded together in 1989 to form the Academic Medical Center Consortium, the purpose of which was to improve the quality of health care through multi-institutional health services research and resulting benchmarking opportunities.

The first project undertaken by the Consortium (and the one that served as the initial impetus for the formation of the Consortium) was an examination of the appropriateness of use of several common procedures in Consortium hospitals. All 12 centers agreed to share in the financial support of the study, make their records available, publish the study results, and take measures to remedy any substantial deficiencies that were discovered. The procedures studied were coronary artery bypass graft (CABG) surgery, aortic aneurysmectomy, carotid endarterectomy, and cataract surgery.

For the study of CABG surgery, an expert panel was convened by the RAND Corporation and the Academic Medical Center Consortium to develop criteria with which to rate appropriateness. These criteria were then used to evaluate the appropriateness of CABG surgery for patients having this procedure done in each of the Consortium hospitals. The Consortium also evaluated the judgment process used by the expert panel to determine appropriateness: Cardiac surgeons from the Consortium institutions reviewed the expert panel ratings and revised them according to consensus. The cases in which CABG surgery had been classified as inappropriate or uncertain according to the RAND-Academic Medical Center Consortium criteria were then reviewed by health services researchers and cardiac surgeons at the Consortium institutions to evaluate the validity of the evaluations and to establish whether the researchers and surgeons agreed with the evaluations.

We report the results of a study that asked four questions. First, how appropriate is the use of CABG surgery in the 12 Academic Medical Center Consortium hospitals? Second, how would appropriateness rates change when the revised criteria developed by the Consortium cardiac surgeons are used? Third, how do assessments of the appropriateness of CABG surgery compare with the implicit judgments by health services researchers and cardiac surgeons of care in their own institutions? Fourth, how do rates of the appropriateness of use of CABG surgery in Consortium hospitals compare with those rates previously found in New York State?

Methods

Top Methods Results Discussion Author & Article Info References

Indications and Criteria for Appropriateness

Criteria for measuring appropriateness were developed by using a previously described method [1, 2]. First, the relevant medical literature was reviewed and synthesized into an annotated summary of the evidence for the effectiveness and the risks of CABG surgery for patients with each of the indications for CABG surgery. Next, a set of clinical scenarios ("indications") was derived that encompassed all of the reasons (both appropriate and inappropriate) for CABG surgery that would be likely to arise in clinical practice. An example of an indication would be class III-IV chronic stable angina in a patient who has been treated with maximal medical therapy, who has three-vessel disease and an ejection fraction greater than 35%, who is a low risk for bypass surgery, and who is not a candidate for percutaneous transluminal coronary angioplasty. Indications were grouped into symptom complexes and syndromes, called "chapters," such as chronic stable angina. A single chapter thus had many indications, each representing a unique combination of essential factors. Chronic stable angina comprised 324 indications. The appropriateness of CABG surgery was rated for each indication both if the patient was and if the patient was not a candidate for percutaneous transluminal coronary angioplasty. The ratings were repeated for each of three levels of surgical risk; a total of 972 indications were rated in the chronic stable angina chapter alone. A sample page of ratings is shown in Appendix Figure 1.

View larger version (35K): [in this window] [in a new window]   Appendix (Figure 1). Sample page of Rand Corporation ratings for the appropriateness of coronary artery bypass graft (CABG) surgery and percutaneous transluminal coronary angioplasty (PTCA) for a patient with severe chronic stable angina. ECG equals electrocardiogram; Pt equals patient.

Panelists were asked to read the literature review and then rate each indication on a nine-point scale as to the appropriateness of CABG surgery for that indication. They were asked to use their best clinical judgment for an average patient presenting to an average surgeon doing CABG surgery in 1990. A procedure was defined as "appropriate" if the net expected health benefit of surgery (quality of life, longevity, or both) exceeded the expected negative consequences (pain, disability, or risk for death) by a margin large enough to make the procedure worth doing. An appropriate procedure is preferable to the alternatives, including other procedures, medication, and no treatment.

Ratings were confidential and were developed in two rounds. The first round was done by mail, the second after face-to-face group discussion. The final appropriateness rating for each indication was the median of the nine panelists' ratings after the second round of ratings. Surgery was considered "appropriate" for a given indication if the median rating for that indication was 7 to 9, "inappropriate" if the median rating was 1 to 3, and "uncertain" if the median rating was 4 to 6. Surgery for a given indication was also considered uncertain if the panelists disagreed about its appropriateness; disagreement was defined as three or more ratings in each of the polar triads (1 to 3 and 7 to 9). In a third round, panelists re-rated indications that had been classified as appropriate for necessity. Surgery for an indication was considered "necessary" if not offering it to most patients with that indication would constitute improper care. Thus, indications for which CABG surgery was classified as necessary were appropriate indications for which CABG surgery should be recommended. All necessary indications were, by definition, appropriate.

The same ratings had previously been used to study the appropriateness of the use of CABG surgery in New York State [3]. The literature review, the listing of all appropriateness ratings, the definitions of terms, and the final ratings by the expert panel of appropriateness and necessity have been published [4].

Revision of Appropriateness Ratings by Cardiac Surgeons

The ratings of the expert panel, the literature review, and the definitions of terms were presented in March 1992 to a meeting of cardiac surgeons from the 12 Academic Medical Centers Consortium institutions. This was done after data had been collected but before the surgeons knew the results of the data analysis. The indications were discussed by chapter, and revisions were developed by consensus. The Consortium surgeons did not change the structure of the indications; that is, they did not add, delete, or modify any of the clinical or laboratory variables that made up the indications.

Data Collection and Sample

We randomly selected study patients from each of the 12 Consortium centers from lists of all patients who had had CABG surgery in 1990. Consecutive records were requested from these lists for data abstraction until 100 records from each institution had been abstracted. Patients who had previously had CABG surgery or concurrent valve replacement surgery were excluded. Patients whose records were excluded or missing were replaced by the next patient on the list. Some patients were not excluded until data collection had been completed, leaving a final sample of 1156 patients. The study was approved by the human subjects committees at each institution in the Academic Medical Center Consortium and at the RAND Corporation.

A medical record abstraction form was created to capture the data needed to determine the appropriateness of CABG surgery. Medical records were abstracted by experienced nurses and abstractors trained to use the form. All abstracted records were reviewed by a supervisor from the RAND corporation for completeness, accuracy, and consistency. Photocopies of the admission note; the discharge summary; results of stress tests, echocardiography, and other noninvasive tests; coronary angiograms; and surgical notes were provided for interpretation by a physician reader, who coded the results of the key tests, reviewed the abstract, and assigned each patient to a specific clinical chapter. To ensure confidentiality of information, coded identifiers were assigned to patients and hospitals. After the data collection process had been completed, the files linking patient identifiers were destroyed.

Analysis

We assigned an indication to each patient according to the information abstracted from the record. If more than one indication applied to a patient, the indication for which CABG surgery was more appropriate was used. To compare the effect of the changes in appropriateness ratings produced by the Academic Medical Center Consortium cardiac surgeons, we did a second analysis using the revised ratings.

Mean rates with 95% CIs were calculated using the normal approximation and were truncated at zero if the approximation extended below zero. Comparisons between two categories were made using relative risks with 95% CIs calculated from bivariate logistic regression results. Differences in distribution across multiple categories were assessed using the chi-square statistic for the unweighted contingency table. Standard errors were inflated using the method of Huber [5] as implemented in Stata [6] to account for clustering effects within hospitals.

Cardiac Surgeons' Review of Cases for which Coronary Artery Bypass Graft Surgery Was Classified as Inappropriate or Uncertain

The records of the patients whose CABG operations had been classified as inappropriate or uncertain according to the RAND-Academic Medical Center Consortium criteria were reviewed by health services researchers and cardiac surgeons in the Consortium member institutions. For each case, the cardiac surgeons were asked to 1) identify any errors in data collection or interpretation, 2) identify critical data that were not available in the medical record but that had been known to the physicians caring for the patient at the time of surgery, and 3) indicate whether they agreed with the appropriateness classification assigned to the case.

Results

Top Methods Results Discussion Author & Article Info References

Of the patients who had CABG surgery, 74% were men; 91% were white, 4% were black, and 1.4% were Hispanic. Twenty-eight percent were younger than 60 years of age, and 16% were 75 years of age or older. Ninety percent of CABG operations were done for indications in one of three clinical categories: chronic stable angina (35%), post-myocardial infarction (28%), and unstable angina (27%) (Table 1). As judged by using the modified Parsonnet score for mortality [4, 7], 54% of patients were in the low-risk group, 33% were in the moderate-risk group, and 13% were in the high-risk group.

According to the RAND-Academic Medical Center Consortium ratings, 92% of the CABG operations were necessary or appropriate (83% [95% CI, 81% to 85%] were necessary; 9% [CI, 8% to 10%] were appropriate), 7% (CI, 5% to 8%) were uncertain, and 1.6% (CI, 1% to 3%) were inappropriate (Table 1).

More than three fourths of the CABG operations were done for either left main (25%) or three-vessel (53%) disease. Only 5% of patients had single-vessel disease. None of the operations for left main or three-vessel disease were classified as inappropriate, but 5% of the operations for two-vessel disease and 14% of the operations for single-vessel disease were classified as inappropriate. Sixteen percent of patients with two-vessel disease and 34% of patients with single-vessel disease had operations for indications that were classified as uncertain Figure 1.

View larger version (27K): [in this window] [in a new window]   Figure 1. Appropriateness of coronary artery bypass graft surgery by vessel. Operations for left main disease (290 patients) or three-vessel disease (613 patients) were more likely to be classified as necessary or appropriate than were those for two-vessel disease (195 patients) or one-vessel disease (58 patients).

Appropriateness did not differ significantly across age categories, except that patients 75 years of age and older were more likely to have surgery classified as uncertain (12%; CI, 9% to 15%) than were younger patients (6%; CI, 3% to 8%). Asymptomatic patients (4% of all patients) were more likely to have surgery classified as uncertain (23%; CI, 10% to 36%) or inappropriate (6%; CI, 1% to 11%) than were other patients (7% of other patients had surgery classified as uncertain [CI, 5% to 8%]); 1.5% of other patients had surgery classified as inappropriate [CI, 0.6% to 2.5%]).

For all 1156 patients, only 395 of 2379 rated indications were actually used in practice. The most frequently used indications for which CABG surgery was judged appropriate, uncertain, or inappropriate are shown in Table 2. Overall, the most common indications for which surgery was judged appropriate were those involving three-vessel disease in low-risk patients. The most frequently used indications for which surgery was judged uncertain were in patients with post-myocardial infarction and three-vessel disease. Cases for which CABG surgery was judged inappropriate were distributed over many indications.

Interhospital Comparisons

The rate of inappropriate CABG surgery among Academic Medical Center Consortium hospitals varied from 0% to 5% (P = 0.02) (Figure 2). Differences in the rates of uncertain, appropriate, and necessary CABG surgery were not significant.

View larger version (37K): [in this window] [in a new window]   Figure 2. Appropriateness of coronary artery bypass graft by hospital. The percentage of cases classified as necessary or appropriate ranged from 85% to 96%. Rates of inappropriate use ranged from 0% to 5%.

Comparison of Expert Panel and Cardiac Surgeons' Ratings for Indications

The revision of the RAND-Academic Medical Center Consortium expert panel ratings by the Consortium cardiac surgeons changed the appropriateness ratings for 568 of the 2379 indications (24%). The appropriateness rating was upgraded (for example, from uncertain to appropriate) for 503 indications and downgraded for 65 indications (Table 3); (Figure 3). The largest number of changes (n = 156) occurred in the post-myocardial infarction chapter, but the chapter with the highest percentage change was evolving acute myocardial infarction (67% of ratings in that chapter changed).

View larger version (19K): [in this window] [in a new window]   Figure 3. Effect of revisions of ratings by surgeons. The number of indications that were upgraded (n = 503) far exceeded the number that were downgraded (n = 65). However, the number of cases affected by these changes was relatively small: When the surgeons' ratings of appropriateness were substituted for the RAND expert panel ratings, ratings were upgraded in 23 cases and downgraded in 27. No case was reclassified from inappropriate to appropriate as a result of the surgeons' revisions.

Three reasons accounted for 69% of the changes in ratings. First, the cardiac surgeons did not believe that candidacy for percutaneous transluminal coronary angioplasty (that is, anatomical feasibility) was relevant to the decision about the appropriateness of CABG surgery. Consequently, they upgraded all ratings for indications for which the rating had been lower if the patient was also a candidate for percutaneous transluminal coronary angioplasty. This alone accounted for 294 of the changes (52%). Second, the surgeons did not consider high surgical risk to be an important mitigating factor in the decision to recommend CABG surgery. Ratings for high-risk patients were upgraded to equal those for low-risk patients for 71 indications (13%). Third, the surgeons considered CABG surgery to be appropriate for almost all patients with evolving acute myocardial infarction, and thus they upgraded ratings for 23 indications (4%). Ratings were downgraded primarily for patients with unstable angina or post-myocardial infarction who had single-vessel and two-vessel disease, were free of symptoms, and had less than markedly positive exercise stress test results.

Effect of Rating Changes on the Analysis of Appropriateness

When the revised ratings were assigned to the 1156 study patients, the appropriateness classifications changed in 50 cases (4%). The classification of appropriateness increased in 23 of these 50 cases and decreased in 27 (Table 4). Thus, although the number of indications that were upgraded outnumbered those that were downgraded by nearly eight to one (503 compared with 65), most of the upgraded indications rarely occurred in practice; the number of cases upgraded was therefore actually fewer than the number downgraded. The number of cases that were changed from uncertain to appropriate (n = 19; 1.6%) was matched by an equal number of cases that were changed from appropriate to uncertain. However, 8 cases (0.7%) were changed from uncertain to inappropriate, whereas 4 cases (0.3%) were changed from inappropriate to uncertain Figure 3. No cases were changed by two categories (from inappropriate to appropriate or appropriate to inappropriate). As a consequence of the four additional inappropriate cases, use of the surgeons' re-ratings resulted in an increase in the overall rate of inappropriate use of CABG surgery from 1.6% to 1.9%.

Local Hospital Review of Uncertain and Inappropriate Cases

Cases for which CABG surgery was classified as inappropriate or uncertain by RAND-Academic Medical Center Consortium criteria were reviewed by local cardiac surgeons in 9 of the 12 Consortium centers. Sixty-four cases were reviewed: 13 of 18 inappropriate cases (72%) and 51 of 76 uncertain cases (67%). This review showed that errors in data abstraction had occurred in 4 cases (6%) and that, in an additional 4 cases, information had been known to the local physicians that was not available to the data abstractors. Correction of these errors and omissions altered the classifications of 3 cases from inappropriate to uncertain (2 cases) or from inappropriate to appropriate (1 case) and altered the classifications of 5 cases from uncertain to appropriate (1 case) or from uncertain to necessary (4 cases). Thus, errors or incomplete data leading to erroneous classifications were identified in 12.5% of patients (Table 5).

In 25 cases (39%; 5 inappropriate and 20 uncertain), the reviewers agreed with the appropriateness classifications. In the remaining 31 cases (48%; 5 inappropriate and 26 uncertain), the reviewers disagreed with the classifications. In 16 of these 31 cases, the reviewers disagreed with the RAND-Academic Medical Center Consortium expert panel definitions (such as the requirement that all candidate vessels have 50% or greater stenosis and [except for left main] that at least one vessel have 70% stenosis). In the remaining 15 cases, the reviewers disagreed with the expert panel rating of appropriateness for the indication. In 3 of these disputed cases, the local surgeon downgraded classifications from uncertain to inappropriate.

Discussion

Top Methods Results Discussion Author & Article Info References

The rate of inappropriate use of CABG surgery in 1990 in 12 academic medical centers was reassuringly low: 1.6%. The rate of uncertain use of CABG surgery was also low: 6.6%. These rates contrast sharply with those found in most other studies that use explicit criteria to determine the appropriateness of use of procedures. One of these was a previous study of CABG done by the RAND corporation from 1979 to 1982, which showed that the rate of inappropriate use was 14% and the rate of uncertain use was 30% [8].

However, the rates we found are similar to those previously reported from a study of CABG in a random sample of hospitals in New York State. That study showed a rate of inappropriate use of 2.4% and a rate of uncertain use of 7.0% [3]. New York has a statewide cardiac surgical data reporting system, and the number of centers in which CABG surgery may be done is controlled. Thus, cardiac surgical practice in New York may not be representative of cardiac surgical practice in the United States as a whole. In addition, the appropriateness of CABG surgery may have increased nationwide during the 1980s as the expanding use of percutaneous transluminal coronary angioplasty reduced the number of patients with single-vessel disease who received referrals for CABG surgery; this had been a major source of inappropriate use of CABG surgery in earlier studies. We found that most patients (78%) had surgery for left main or three-vessel disease, conditions for which CABG surgery has been shown to have efficacy in randomized, controlled clinical trials [9, 10].

An important question in our study was whether surgeons who did the CABG procedures would find the appropriateness criteria and the application of these criteria to specific cases acceptable. The Academic Medical Center Consortium surgeons did in fact change many of the appropriateness ratings when they reviewed them. However, their changes did not appreciably affect the proportion of patients whose operations were classified as appropriate, uncertain, or inappropriate, because few of the changes that they made applied to indications for CABG surgery that actually occurred frequently in practice. For example, because they believed that CABG surgery was almost always more appropriate than percutaneous transluminal coronary angioplasty for patients needing revascularization, the surgeons gave the same appropriateness rating to CABG surgery for an indication regardless of whether the patient who had the indication was a suitable candidate for percutaneous transluminal coronary angioplasty. This accounted for 52% of the changes made by the surgeons to the expert panel ratings. However, because most of these changes applied to patients with single-vessel disease, a group that made up only 5% of the patients having CABG surgery, these changes had little effect on the rates of appropriate use of CABG surgery.

The second largest category in which ratings were changed was that of patients with high surgical risk, who were given the same ratings as low-risk or moderate-risk patients by the Consortium surgeons. However, high-risk patients were also a small percentage (13%) of the total, and the original expert panel ratings for the most frequently used indications (patients with left main and three-vessel disease) also varied little with surgical risk. Thus, these changes had little effect on the results. Finally, ratings were changed for two thirds of the indications for CABG surgery in patients with acute myocardial infarction, but these patients made up fewer than 1% of the total patients.

In evaluating the surgeons' revisions of the appropriateness ratings, it is important to note that the cardiac surgeons were not convened as a group de novo for the purpose of developing and rating indications for CABG surgery but were responding to the RAND-Academic Medical Center Consortium ratings. However, they did so after considerable study of those ratings, including review of extensive explanations of the structure and definitions of those ratings (which they did not change). It is also noteworthy that they made the changes before they knew the results of the study, either overall or for their own patients. In addition, the re-rating occurred 16 months after the expert panel rating was done, and this period was characterized by increasingly aggressive care for patients with coronary artery disease.

The surgeons' revisions reflect the predilection of specialists to favor their own procedures (that is, they preferred CABG to percutaneous transluminal coronary angioplasty for all patients). It has been shown that physicians who do procedures find the use of those procedures appropriate for more indications than do physicians who do not do the procedures [11]. Specialists also tend to focus on the positive aspects of their work and downplay risk factors. Thus, the surgeons' revisions of the ratings illustrate the types of changes that might be expected to occur if hospital specialty physicians were given the opportunity to modify national multispecialty guidelines for their own use, as has been suggested by some as an important part of the "buy-in" process. On the other hand, the bias of specialists is precisely the reason the "balanced" panel method was developed to include input by generalists and other specialists who do not do the CABG procedure. Disinterested parties, for example, do not downplay alternative therapies or the importance of surgical risk. It is pertinent that in a recent study of the reproducibility of this panel process, none of three independent balanced panels judged CABG to be preferable to percutaneous transluminal coronary angioplasty for most patients with single-vessel disease or altered the reduction in appropriateness associated with higher risk (Shekelle P. Personal communication).

Review of the hospital records of patients who had CABG surgery that was classified as inappropriate or uncertain for their indications showed missing information or data collection errors in 12.5% of the records. A similar rate of errors and omissions would probably be found if the cases classified as appropriate or necessary were reviewed. Because these cases account for 92% of all cases, one can speculate that identification of all errors might result in an increase in the overall rate of inappropriate and uncertain use of CABG surgery.

In practice, these rates may have little importance. Appropriateness ratings have found their greatest use as prospective instruments with which to identify cases of potentially inappropriate use of procedures. These cases are then reviewed by the physician, who supplies missing data and corrects any data errors. Even when the criteria are used as a screening technique in a retrospective audit, identified cases are typically subjected to physician rereview. Our data suggest that 10% to 15% of cases so reviewed would be upgraded as a result of data errors.

As one would expect, when hospital surgeons were asked to review their own cases, they often disagreed with the negative classifications given by the expert panel. We found that the judgment of the hospital surgeons differed from that of the expert panel for half of the patients whose operations were classified as inappropriate or uncertain. On the other hand, the surgeons agreed with the findings of the panel in 39% of the cases in which CABG surgery was classified as inappropriate or uncertain. In addition, for three cases (5%) about which the surgeons disagreed with the expert panel, the surgeons actually downgraded the classification from uncertain to inappropriate. Such differences are not surprising given the variations in clinical practice and the tendency of even highly skilled and objective physicians to view their own cases differently than would an external party.

Implications for Policy

Because of the large number of bypass operations that are done and the risks and costs associated with them, ensuring the appropriate use of CABG surgery has been of interest for some time both to payers and to persons concerned with quality of care. An important question for policymakers is, How can practitioners be motivated to adopt best practices? [12].

One way to do this is to provide practitioners with data from outcome studies. Studies of CABG surgery are exemplary. Perhaps no operation in all of medicine has been the subject of so many randomized, controlled studies. As a result, evidence is available about the risks and benefits of CABG surgery for many indications. This may be a major reason the rate of inappropriate use of CABG surgery is so low.

Another way to improve practice is to give feedback to physicians about the quality of their patient care. Methods for measuring quality of care have greatly improved in recent years; both process and outcome measures are far more sophisticated than they were just a decade ago. The appropriateness method is a new process measure based on evidence of effectiveness and expert judgments of probable outcomes. But, like any information about quality of care, information on appropriateness must be presented to physicians in a meaningful, patient-specific, clinical context so that the need for changes in practice can be recognized.

In our study, the member institutions of the Academic Medical Center Consortium undertook a self-examination that combined external assessment with internal review of both criteria and results. Although there are clearly powerful reasons physicians would be motivated to reject any classification of their cases as "inappropriate" or "uncertain," we found that after data errors were eliminated, the Academic Medical Center Consortium cardiac surgeons were willing to accept the judgment that CABG surgery was uncertain or inappropriate for the indications used in nearly half of the cases for which the use of CABG surgery was classified as uncertain or inappropriate.

Our findings also show that even rigorously derived appropriateness criteria should not be applied blindly in the evaluation of individual patients. If used in audits, appropriateness criteria should be used to identify cases for further review by physicians. They should not be used to label cases of use as inappropriate or uncertain, because the ratings cannot anticipate all extenuating circumstances that might apply to an individual patient. Therefore, noncompliance does not imply either that care was poor or that the criteria are invalid. On the other hand, for group estimates of appropriateness, it is apparent that neither revision of the ratings nor local physician review are likely to change results substantially or invalidate comparisons between institutions or regions.

Use of Appropriateness Criteria

Any hospital can use explicit criteria to appraise the appropriateness of use of CABG surgery as the Academic Medical Center Consortium institutions did, either prospectively or retrospectively. Prospectively, measures of appropriateness can be computerized so that physicians could access them during decision making. Retrospective use can identify problem indications or physicians with patterns of inappropriate care.

Motivated by this study and by their interest in improving the quality of patient care, the cardiac surgeons and cardiologists of the Academic Medical Center Consortium began collecting data from all patients who had CABG surgery or angioplasty at the member institutions in July 1993. They are comparing both process and risk-adjusted outcome variables to learn from one another how to improve patient care.

Conclusions

Academic medical centers have low rates of inappropriate and uncertain use of CABG surgery. These rates are similar to those reported from a population-based study in New York State. Although cardiac surgeons in the Consortium hospitals made substantial revisions in the appropriateness criteria, when their ratings were applied to the same patients, the effect of the revisions on the overall assessment of appropriateness was negligible.

Local review of cases in which the use of CABG surgery was classified as inappropriate or uncertain showed an error rate of 12.5% due to missing data or data collection errors. Although the overall error rate might be lower if errors made in cases classified as appropriate canceled out some of those in the cases classified as uncertain or inappropriate, such errors would probably have little importance in comparisons of group estimates of appropriateness. For individual case audits, whether prospective or retrospective, it is clear that appropriateness criteria should be used only as a first step to be followed by individual physician review. This practice is, in fact, routinely followed by responsible utilization review organizations.

Appendix

This study was part of the Clinical Appropriateness Initiative, a cooperative effort among the RAND Corporation, the Academic Medical Center Consortium, and the American Medical Association.

The member institutions of the Academic Medical Center Consortium are Brigham and Womens' Hospital (Boston, Massachusetts); Dartmouth-Hitchcock Medical Center (Lebanon, New Hampshire); Duke University Medical Center (Durham, North Carolina); Johns Hopkins Hospital (Baltimore, Maryland); Massachusetts General Hospital (Boston, Massachusetts); Mayo Foundation (Rochester, Minnesota); New England Medical Center (Boston, Massachusetts); Alton Ochsner Medical Foundation (New Orleans, Louisiana); University of California, Los Angeles, Medical Center (Los Angeles, California); University of Iowa Hospitals and Clinics (Iowa City, Iowa); University of Pennsylvania Health System (Philadelphia, Pennsylvania); and University of Rochester Medical Center (Rochester, New York).

The members of the Working Group of the Appropriateness Project of the Academic Medical Center Consortium are Robert H. Brook, MD, ScD, Chair, and Lucian L. Leape, MD (RAND Corporation, Santa Monica, California); David M. Witter Jr. (Academic Medical Center Consortium); John Kelly, MD (American Medical Association, Chicago, Illinois); Edward Bluth, MD, and Tonette Krousel-Wood, MD (Alton Ochsner Medical Institutions); Anthony Komaroff, MD, and David W. Bates, MD, MSc (Brigham and Women's Hospital); David Matchar, MD (Duke University Medical Center); Haya R. Rubin, MD, PhD (Johns Hopkins Hospital); Peter Slavin, MD (Massachusetts General Hospital); David Ballard, MD, PhD (Mayo Foundation); Katherine Kahn, MD (University of California, Los Angeles, Medical Center); Joanne Tobacman, MD (University of Iowa Hospitals and Clinics); J. Sanford Schwartz, MD (University of Pennsylvania Medical Center); and Robert J. Panzer, MD (University of Rochester Medical Center).

Dr. Bates: Department of General Medicine, Brigham and Womens' Hospital, 75 Francis Street, Boston, MA 02115.

Drs. Brook, Hilborne, and Park: The RAND Corporation, 1700 Main Street, Santa Monica, CA 90406.

Dr. Panzer: University of Rochester Medical Center, 601 Elmwood Avenue, Rochester, NY 14642.

Dr. Rubin: Johns Hopkins University Hospital, 1830 East Monument Street, Baltimore, MD 21205.

Dr. Schwartz: Leonard Davis Institute, 3641 Locust Walk, Philadelphia, PA 19104.

Dr. Slavin: Center for the Evaluation of Medical Practice, Massachusetts General Hospital, 50 Staniford Street, Boston, MA 02114.

Mr. Witter: Academic Medical Center Consortium, 30 Corporate Woods, Suite 300, Rochester, NY 14643.