Recognizing the need for reform, the Health Care Financing Administration (HCFA) has proposed a new direction for the PROs' reviews of quality: the Health Care Quality Improvement Initiative. The primary goal is to replace the punitive model of peer review with one that emphasizes education and continuous quality improvement [7, 8]. A major change is that the predominant form of review in the PRO, the case-by-case review, will be for the most part abandoned and replaced by profiles of practice patterns. Individual physicians will no longer be routinely scrutinized on a case-by-case basis to find clinical errors. Instead, patterns of care of groups of physicians (for example, the percentage of diabetic patients referred for yearly eye examinations) will be monitored at the institutional, regional, or national level. Such information will then be disseminated within an environment that stresses professionalism, to be used by providers to solve problems and improve the quality of the care they deliver.

The development of an accurate and valid database is an essential component of this initiative. According to the HCFA, the Uniform Clinical Data Set (UCDS), the proposed national database for the Medicare peer review program, should solve many of the issues of data accuracy, reliability, and validity as well as meet the data requirements for profiling practice patterns. But is this truly so? We pose questions about whether the UCDS can improve the current PRO review process, how the data that it provides will be used to assess quality, and whether this will be an improvement over quality assessment as it is now done. We recommend realistic roles and expectations for a national data set for reviews of quality within the larger context of quality improvement activities occurring at regional and local levels. The UCDS is still in a pilot phase, and these issues need to be resolved before its scheduled nationwide implementation in 1996.

Definition and Role of the UCDS in the Proposed New Medicare Quality Review Program

The UCDS is an automated, computerized data set designed to standardize the evaluation of the quality of care that Medicare beneficiaries receive. The UCDS should allow an objective, consistent, and efficient process for peer review based on explicit decision criteria and on aggregated information about patterns of care and quality.

Its development began 5 years ago through joint efforts by the HCFA, PROs, and clinical researchers. The UCDS can currently accommodate 1800 clinical variables per patient. The clinical data will be abstracted from medical records and classified into 10 categories: 1) patient identification; 2) history and physical examination; 3) laboratory results; 4) diagnostic test results; 5) endoscopic procedures; 6) surgical interventions; 7) therapeutic interventions; 8) in-hospital medications; 9) recovery phase; and 10) discharge planning and discharge status. It is expected that approximately 200 to 600 elements per case reviewed will be abstracted.

The UCDS should contribute to improvements in two areas of quality assessment: "generic quality" Table 1 and "disease-specific" quality reviews. Generic quality reviews consist of the random screening of a 5% sample of all Medicare admissions for problems that are independent of diagnosis, such as the patient's clinical stability at the time of discharge. By computerizing the current manual data collection, the UCDS is expected to increase the accuracy and the reliability of the review process. The criteria used to evaluate generic problems of quality are also being revised, and this may improve the validity of the review process.

The disease-specific reviews of care focus on defined clinical conditions, such as the management of patients with acute myocardial infarction. The review process currently consists of individual case reviews. The UCDS is expected to replace this inefficient method with reviews based on the analysis of large numbers of cases. Such analyses could suggest where potential problems exist and permit a more appropriate focus for further reviews. Compared with traditional isolated reviews, this approach, the profiling of patterns of care (for example, the use of thrombolytic agents in the management of acute myocardial infarction), could help lead to improvements of quality of care at national and regional levels as well as at the level of the individual practitioner. In the future, generic reviews of quality—the random, case-by-case method of review—should gradually be replaced by reviews of disease-specific patterns of practice, as the capacity to measure the quality of care of an increasing number of clinical conditions continues to grow.

Accuracy, Reliability, and Efficiency of Data Abstraction

Before the UCDS data can be used to evaluate care, their quality must be ensured. A good instrument should abstract data accurately, reliably, and efficiently. In the current peer review program, evidence suggests that data abstraction is deficient [9-11]. Will the UCDS be in any way superior?

As in the current PRO system, medical chart audits will continue to be the main source for abstracting data that will be input in the UCDS (Tables 2 and 3). Nurses are usually responsible for the initial audits, from which they determine whether the case warrants further review by a physician. Most studies of data abstraction pertain to generic quality reviews [3, 7, 9-12], and initial nurse abstraction has been shown to be inaccurate and unreliable (Table 3). Similar data on the UCDS have not been generated, and, although computerizing data abstraction should improve accuracy and reliability, this has not yet been proved. The UCDS must be formally evaluated for criteria such as inter-rater and inter-item reliability, especially because a large number of abstractors will be collecting data from diverse health care environments.

In addition to accuracy and reliability, the overall efficiency of a data collection instrument requires consideration. One objective for the UCDS is the automation of the abstraction process to reduce time and expense. Preliminary results from PROs doing pilot studies of the UCDS (January 1992) have not been encouraging. The average time to review a chart using the UCDS was 93 minutes. These figures contrast with the average 23 minutes for a nurse to review a medical record manually in the standard way (Health Care Financing Administration. Office of Peer Review. Health Standards and Quality Bureau. Baltimore, Maryland. April 1992). These preliminary findings suggest that methods of data abstraction need to be improved or fewer data gathered so that their collection is timely and cost-effective. Otherwise, the system will be too expensive to sustain.

Once data have been accurately and reliably abstracted, they will be analyzed to evaluate quality. The key issue here is validity (that is, the data and the methods of review should evaluate what they are meant to evaluate: quality). We need to ask whether the UCDS can increase the validity of the generic quality and the disease-specific review process.

Validity of the Generic Quality Reviews

Several studies have evaluated the validity of the current PRO generic quality reviews [9-12]. Results have been poor, both for the initial screening process by nurse reviewers and for the steps that follow—physicians' reviews and final decisions. In one study, the PRO's judgment of quality agreed with the judgment based on explicit criteria only slightly more often than predicted by chance [3]. The initial screening by nurses was shown to be nonsensitive and nonspecific in identifying quality problems. The initial screen failed to identify two of three records that were later judged to represent quality problems. Physicians' reviews and final judgments were also found to be inadequate. This was attributed to the "implicit" nature of the review method, in which physicians use their own ad hoc criteria to judge quality [12].

Will the UCDS solve these problems of validity? To do so, the rules of the generic reviews will need to be highly sensitive in detecting quality-of-care problems as well as highly specific (that is, a problem-free admission should not be inappropriately categorized). In addition, the reviews will need to be valid predictors of quality so that cases flagged for physician review have a high likelihood of representing true quality problems (high positive predictive value) and true quality problems do not escape this initial screen (high negative predictive value).

Few data currently exist on the sensitivity, specificity, and predictive value of the revised UCDS generic screens. The data that do exist have been disappointing and suggest that much work is needed before the UCDS reaches the level of performance required of an instrument used to evaluate quality of care nationally. In the phase I pilot test of the UCDS conducted in seven PROs, 21 312 cases were reviewed by both the traditional nurse review process and the new UCDS system. The rate of cases referred for physician review (for utilization and for quality problems) by the UCDS system was slightly higher (51%) than that by the traditional method (45%) (Health Care Financing Administration. Office of Peer Review. Health Standards and Quality Bureau. Baltimore, Maryland. April 1992). The proportion of cases referred for quality review that were defined as quality problems after physician review, however, was similarly low for both methods (6%).

In the pilot study, physicians continued to use implicit review criteria, a method that has been shown to be flawed [12]. Until an explicit or a structured implicit method of review [12] is introduced in the process [13], the subjectivity of implicit reviews will continue to compromise validity.

Disease-Specific Profiles of Practice Patterns: Potential for Quality Improvement

The second method that will be used for reviews of quality is the statistical analysis of disease-specific practice patterns. The underlying principle is that information obtained through pattern analysis can enhance quality-of-care initiatives. How can the data from the UCDS best be used to achieve these objectives?

We propose the use of a stepwise approach to profiling for reviews of quality (Table 4). In this model, each step provides an increasing degree of detail about the aspect of care being studied. Data requirements increase at each level and the ease of data acquisition decreases. The appropriate locus for the review (national, regional, or institution-based) is also determined by the type of review. For instance, surveillance (the first level of review) is best suited for reviews at regional and national levels. But the next steps, generating hypotheses, confirming or rejecting the existence of quality problems, and identifying their causes, are best suited to local internal quality improvement programs.

Where does the UCDS fit in the context of this model for quality reviews? In the example we use in Table 4, the review of quality related to cesarean section, surveillance is done by reviewing the cesarean section rate for the nation, for states, for regions, and for institutions and health plans within these regions. Such information can easily be included in a database such as the UCDS where it would be readily available.

Surveillance might reveal that a particular region has an unusually high rate of cesarean sections. At this point, several hypotheses can be generated. Perhaps physicians in that area were unaccustomed or unwilling to attempt a vaginal delivery after a previous cesarean section. By gathering data on the rates of primary and repeat cesarean sections, this hypothesis can be explored, although it requires more detailed record abstraction. If the region had a high rate of repeat cesarean sections, these data would support but would not prove that a problem existed or identify its causes. To find this out, additional, more specific data would be needed that are beyond what a national database, even one as sophisticated as the UCDS, could realistically provide. At this stage, evaluation of the quality and the appropriateness of care would need to include the active participation of those involved in the delivery of care. Medical record data may not be sufficient, and a survey of patients' knowledge and values or an analysis of system-level processes of care may be necessary.

In summary, this model of quality reviews suggests that the principal role for national databases such as the UCDS should be confined to surveillance [14]. Given this role, profiling can only raise questions; it cannot answer them. The evaluation of care must be assessed with much more detail to confirm that a problem exists and to determine its causes. Nationwide profiling should not be used to judge complex clinical problems; it is unrealistic to expect the UCDS to capture the nuances of clinical care from medical records, which are not and never will be perfect data sources. Thorough assessments will ultimately require information that can only be obtained by listening to and learning from those directly involved in the delivery of care. This approach to reviews of quality offers the opportunity to actually improve quality of care and to instill new momentum to peer-review activities long mired in complex data sets and implicit, often secret, review criteria. Initiatives to improve quality can only succeed if they are credible to practicing physicians. The credibility of the PRO quality review program depends on the current efforts to develop reliable and valid criteria and processes by which to evaluate quality of care and to define practice environments that are conducive to quality improvement activities.

Conclusion

Top Conclusion Author & Article Info References

Peer review of medical care has always occurred, in one form or another, in various clinical settings [15-20]. It is a fundamental part of medical practice—sometimes formal, sometimes informal—that contributes to enhanced knowledge and improved care. As we seek state-of-the-art methods to promote the delivery of cost-effective and quality care, information about quality becomes pivotal. This information must be accurate, reliable, and valid to be credible and to translate into effective decision making and improved outcomes. As a national database, the UCDS could respond to some of those needs and become an important source of information about the delivery of health care in this country.

The greatest strengths of the UCDS still lie in what it promises, and to achieve such goals the system will require further improvements. Continued national support for the UCDS requires strong evidence about how and if expectations are being met. We again emphasize that not enough is known about the UCDS to enable the medical community to judge it. It is essential that objective data about its performance be made available before it is implemented nationwide.

Presenting objective data about the current PRO quality review process and comparing it with the new direction proposed by HCFA is both important and timely. Given the role that UCDS is expected to play as the database central to the Medicare quality assurance program, the HCFA must ensure that it be of high standards. The UCDS can play an important role in quality review programs, but this role can only be limited, relatively specific, and circumscribed in the broader context of quality improvement activities occurring at local and regional levels. The medical profession must continue to provide guidance and leadership in the development and implementation of the UCDS to ensure its harmonious integration in peer review and quality-of-care initiatives.