Little information exists on the accuracy of medical records as the basis for quality-improvement efforts. Although we have previously shown that most examples of malpractice can be found in medical records [8], nagging doubts remain that evidence of poor quality may not be documented in the record by providers. Retrospective record review also tends to make providers feel as though they are being investigated and may reduce physician interest in quality assurance. Finally, review of records may be expensive [9].

A different method of gathering data for quality-improvement purposes is to ask physicians and other providers to concurrently identify patients who received substandard care. A physician reporting system obviates the weakness of relying on the medical record. It can also be more easily integrated into a philosophy of quality improvement, which many have argued is essential for real change [10, 11]. "Time-of-event" identification of problems offers greater potential for mitigating them. In addition, the identification of quality problems as they occur may be less expensive than retrospective record review. Of course, given physicians' negative attitudes toward quality-improvement efforts, many might doubt the feasibility of instituting an effective physician reporting system.

Because of the theoretical advantages of physician identification of quality problems, we compared physician reporting of adverse events by the housestaff on the medical service of a teaching hospital with a retrospective two-step screening of the medical record for adverse events followed by implicit physician judgment about identified events. We determined the relative efficacy of the physician reporting versus the chart-review screening processes and determined whether the two processes identified the same adverse events. We also compared both of these approaches to the traditional quality-assurance devices used at this hospital, which are similar to those of other hospitals. We report the results of our comparisons and comment on their implications for new trends in quality improvement.

Methods

Top Methods Results Discussion Author & Article Info References

Sample

We studied all 3146 admissions to the medical service of the Brigham and Women's Hospital for the period 13 November 1990 to 14 March 1991. We were unable to obtain medical records for five admissions. Therefore, the following description is based on the review of 3141 admissions.

Review Techniques

As in previous studies, we used the adverse event as the outcome measure. An adverse event was defined as an injury that prolongs the hospital stay or leads to disability at the time of discharge, which is caused by inappropriate medical management instead of the disease process. We also identified the subset of adverse events that were preventable. Two parallel processes or strategies were used to identify adverse events: a medical-record review strategy (strategy 1) and a physician reporting strategy (strategy 2). In addition, we compared the combined strategies with the quality-of-care data routinely collected by the hospital.

Strategy 1: Medical-Record Review

The first strategy entailed a review of the medical record. All records were initially reviewed by medical-record analysts using a list of screening criteria similar to the one used in the Harvard Medical Practice Study [12]. The 10 medical-record analysts, all fourth-year students enrolled in the Health Information Administration Program at Northeastern University, were familiar with medical-record coding, terminology, and management. All the analysts read a training manual that summarized the chart-review portion of the study and that contained examples of how each of the 15 screening criteria should be evaluated in the context of the study protocol. These screens were designed to signal events in patient care that might be associated with an undesirable outcome that resulted from medical management. We have previously discussed the reliability and validity of these screening criteria [13]. In addition to the 15 screening criteria, medical-record analysts were instructed to collect demographic characteristics such as date of birth, sex, and race.

All charts from patients for whom at least 1 of the 15 study criteria was positive were referred for physician review. Eighteen senior medical residents from another Boston teaching hospital conducted an independent review of the medical records that were screened positive by medical-record analysts. Using the Preventable Medical Injury Form as a guide, physicians reviewed the medical admission for potential adverse events. The Preventable Medical Injury Form focuses the implicit judgments of reviewing physicians on the critical issue of presence of an adverse event (medical causation). Each reviewer was trained in the use of this form during a 2-hour session and was given a training manual that contained directions and that detailed examples of confirmed case-patients.

In the analysis of a case-patient who had positive screening criteria, the physician-reviewers first made a determination about medical management causation. Next, they were asked to classify the event into three categories: 1) procedure-related injury; 2) medication-related injury; and 3) other therapy-related injuries. After the categorization of the injury, there were questions about where the event occurred and the specialty responsible for the injury. The extent and nature of the patient's disability were then assessed. If more than one medical injury occurred during a given admission, physician-reviewers were instructed to summarize only the most disabling or the injury causing the longest additional length of stay. Only one adverse event per patient was counted.

To evaluate the inter-rater reliability of the critical judgments of causation by physician-reviewers, we randomly selected a 10% sample of the medical records that were screened positive by medical-record analysts. Physician-reviewers, blinded to the results of the first screening, were asked to re-review the sample.

Strategy 2: Physician Reporting

The second identification strategy involved the confidential reporting of potential injuries by the medical housestaff. Thirteen medical teams were responsible for reporting events at any one time. The hospital has a heavily used electronic mail system available on 2000 terminals in all areas of the hospital. Senior or junior residents heading each of these teams were reminded daily, using electronic mail, to report events. Case-patient summaries and relevant study elements could be reported to the study team by electronic mail or could be written and left in a study box that was checked daily. Most participating residents elected to report cases using electronic mail. The reporting focused on any patient under the team's care; if the patient was being crosscovered by a physician from another team, the original team was alerted to the adverse event by the crosscovering physician. Thus, the reporting was "physician self-reporting," not reporting on the adverse events of others. During weekly resident meetings, the entire medical housestaff were reminded of the importance of reporting potential injuries. Medical interns were not asked to report potential injuries. Each case-patient for whom an adverse event was reported by the housestaff was then matched to two controls, based on bed occupancy at the time of event. Patients occupying the beds adjacent to the case-patient at the time of the event were selected as controls. The case–control portion of this investigation has been defined in our laboratory [Petersen LA and colleagues. Unpublished data]. After daily reports of potential injuries, resident physicians reviewed the charts of case-patients to determine whether an adverse event had occurred. They were trained in the definition of an adverse event in a manner similar to the physicians who participated in the medical-record review strategy. The type of event was again classified into three categories: 1) procedure-related injury; 2) medication-related injury; and 3) other therapy-related injuries. The location of the event was also noted. Generally, cases were reported within 24 hours of occurrence, and pertinent data elements were completed within 48 hours.

Preventability of Adverse Events

For all adverse events identified by both strategies, judgments of preventability were later made by two senior physician investigators, blinded to all the data elements except detailed, clinical case summaries. The reviewers, working independently, made an initial judgment of preventability on a six-point scale, ranging from little or no evidence of preventability [1] to virtually certain evidence of preventability [6]. For cases in which reviewers differed by more than two points, they met and reached a consensus. For the analysis, we collapsed the scale into dichotomous outcomes (preventable, not preventable). Those case-patients with scores of four or more were considered preventable.

Costs of Review Process

Cost estimates were calculated by multiplying the number of hours worked by the designated hourly wage. Medical-record analysts were paid an hourly base rate of $8.24. Physician reviewers working in the record room were paid $30.00/h, whereas physicians completing data collection about cases and their controls were paid $25.00/h. Residents were not paid to answer daily prompts to report potential adverse events. No fringe benefits were paid to any of these persons. The cost for retrieving medical records for strategy 1 was calculated based on the hourly estimates for retrieving records from storage, tracking, and refiling 3141 medical records. Data-entry costs for strategy 1 were calculated based on the double-entry of all data forms. The Medical-Record Review Form was keypunched for $0.30, the Preventable Medical Injury Short Form (no adverse event present) for $0.22, and the Preventable Medical Injury Long Form (adverse event present) for $1.05. Data entry for strategy 2 was completed by a research assistant who was paid $10.00/h to keypunch all the data for strategy 2. Administrative costs mainly included the part-time salary of a project director for 1 year.

Other Hospital Quality Data

We also reviewed quality data that are compiled in a concurrent manner by the hospital administration. First, the department of medicine identifies potential case-patients or outcomes associated with substandard medical care in the following manner. Nursing professionals screen patients as positive for one of the following: 1) readmission within 15 days of discharge; 2) unplanned transfer to an intensive care unit; and 3) deaths within 24 hours of admission. Charts from patients who screen positive for one of the three screens are then reviewed by two staff physicians who oversee the department's quality-assurance work. These physicians determine whether a quality problem has occurred.

Second, the hospital's risk management program has been responsible for tracking adverse outcomes on a hospital-wide basis for many years. Department personnel learn about adverse events in various ways, but potential case-patients are typically detected by incident reports, by physician and nursing phone calls, and by individual patient complaints. Each of these events is investigated, and a small number are deemed quality problems and are reported to the hospital's professional liability insurer. As with the quality-assurance records, the set of hospitalizations eligible for review was equivalent to that evaluated in the record-review and the physician reporting strategies.

Third, a large health maintenance organization (HMO) admits most of its patients to our hospital. The medical director of the HMO directs a wide-ranging quality-assurance program that includes ongoing identification by liaison nurses of potential quality problems, including medical injuries. These potential cases are reviewed by the medical director, who then decides whether a quality problem has occurred. A total of 806 patients were admitted during the study period from the HMO, each of whom were also evaluated by the record review and the physician reporting strategies.

Finally, the directors of the cardiac catheterization and gastrointestinal endoscopy laboratories both have their own quality-assurance programs. They identify complications from reports by fellows, review the complications, and determine those that represent quality problems. A total of 207 medical inpatients had endoscopy and 644 patients had cardiac catheterization during the study period. Again, each of these cases was evaluated by the record review and physician reporting strategies. For each of the hospital quality-initiative strategies, the senior investigator reviewed all potential quality problems to determine those that were adverse events.

Analytic Methods

To test the comparability of the two study groups, categoric variables were analyzed using the chi-square test or using the Fisher exact test when the expected cell frequency was less than or equal to five. For continuous variables, we used the Wilcoxon rank-sum test because the variables were not normally distributed. Because the coefficient based on the interclass correlation has unwanted constraints when the underlying adverse event rate is not 0.5 [14], we based our coefficients on the odds ratio [15].

Results

Top Methods Results Discussion Author & Article Info References

Comparison of Case-Patient Identification by the Two Strategies

The demographic data for the entire cohort of 3141 medical admissions are listed in Table 1. The mean length of stay for a medical admission during the study period was 6.86 days (SE ± 9.6 days). The mean age was 55 years (range, 15 to 91 years).

The confidential reporting method used by the medical housestaff (strategy 2), identified 124 case-patients. However, when the trained resident physicians independently reviewed the medical record, they determined that 35 patients did not meet the case definition of an adverse event, leaving a total of 89 patients (2.8%). Although the two strategies identified approximately the same number of adverse events during the study period, only a few of the patients were identified by both strategies ( = 0.52; CI, 0.47 to 0.57) (Table 2). Of the 48 adverse events identified by reporting physicians but not found in the record review, 14 failed to fulfill the screening criteria of the medical-record analysts.

Next, we examined the possible socioeconomic and clinical differences between the patients falling into the discordant cells of Table 2. This comparison did not show any statistical differences with respect to age, sex, race, or insurance status. No statistical differences were identified between the two strategies with regard to type of event, location of the event, team responsible, or omission/commission nature of the event leading to injury (Table 3). The housestaff were, however, more likely to report preventable adverse events than were the physician-reviewers during their review (P = 0.003), and housestaff were more likely to report errors of commission (P = 0.04) (Table 3).

The 41 patients identified by both strategies were similar for all of these variables to the 92 identified by only one strategy. For instance, 58% of the patients identified by both strategies were women, 76% were white, 32% were Medicare recipients, and 27% belonged to an HMO. The mean length of stay for patients having adverse events identified by both strategies was 15 days; 61% of those having adverse events found by both strategies had minimal disability, and 39% of the adverse events were preventable.

Nor did close inspection of the clinical details of case-patients show any distinguishing features that were statistically different between the type of patient identified using the different strategies. For instance, the physician-reporting strategy alone found the case of a woman with the acquired immunodeficiency syndrome who was admitted with fever and chills. Her chest radiograph was initially misread as normal, and a delay occurred in treatment of her bacterial pneumonia. The physician report alone also identified Clostridium difficile colitis in a patient taking antibiotics. On the other hand, medical-record review alone indicated that hyponatremia and congestive heart failure developed in a patient after overly aggressive hydration. Both strategies found an elderly woman who had respiratory distress after receiving sedation for a lumbar puncture, and both found a femoral artery aneurysm after a diagnostic procedure. All of these adverse events increased the duration of hospitalization.

Using the hospital's fiscal database, we also examined all inpatient charges for the study period. Our analysis showed no differences between the two discordant groups with respect to pharmacy average charges ($2729 for strategy 1 compared with $2961 for strategy 2; P > 0.2), intensive care unit charges ($5037 compared with $12 665; P = 0.13), room charges ($8207 compared with $5629; P > 0.2), or total charges ($27 204 compared with $29 236; P > 0.2) even after controlling for discharge.

Costs of the Two Review Strategies

For strategy 1 (screening followed by physician structured review), the total time commitment required of the department of medical records to retrieve 3141 charts was 532 hours, which cost $7148. The medical-record analysts salaries were $15 133. Physician review costs were $14 880 for the record review and $6935 for the physician follow-up on the reports. The data entry costs were $2351 for the record review and $338 for the physician reporting. Administrative costs were $14 950 for the record review and $8050 for the physician report. The total for the record review was $54 462 compared with the total for the physician report of $15 323. The latter does not include the costs for the time of the physicians to report on adverse events.

Hospital Quality-Assessment Strategies

The five conventional hospital quality-assessment strategies varied considerably in their efficacy when judged by the number of patients with adverse events who were found using the combined record review and housestaff report strategies (all patients identified by either or both of strategies 1 and 2) (Table 4). Some of the most typical strategies for quality assurance were ineffective for identifying adverse events. For instance, the departmental quality-assurance process identified only 13 adverse events, and the risk management team identified only 4. The endoscopy team found 2 adverse events, and they missed 6. The cardiac catheterization team missed 18 adverse events found by the combined record review and physician report strategies but identified an additional 10 that were not reported using strategy 1 or 2. The combined approach of the housestaff and record review strategies failed to identify many of the HMO patients with adverse events because most of the latter were related to outpatient management issues that led to hospitalization and, hence, fell outside the study's incidence criteria. Overall, the agreement between the conventional approaches and the combined strategies was poor, with the exception of the catheterization laboratory ( = 0.46).

Discussion

Top Methods Results Discussion Author & Article Info References

Our primary hypothesis was that physicians' reports of adverse events would be as effective as a standard record review method. The data show the efficacy of a physician reporting system for finding potential quality problems on a medical service of a teaching hospital. Physicians identified almost the same number of adverse events as were found using a chart-review process based on screening followed by implicit judgments made by physicians. Moreover, physician-reported cases were more often found to be preventable and thus more likely to be affected by quality-improvement efforts.

Perhaps more importantly, the intervention shifted the paradigm from one in which providers were targets of outside scrutiny into another in which they were active participants in quality assurance. In our discussions with housestaff, they did not think they were being asked to "snitch" on one another. Rather, the house officers believed they were being asked to assess honestly the care they and their colleagues were providing. They also believed that the information would be used to improve the quality of care. We consider this change, in physician orientation to quality of care, critical to real improvement. The housestaff are a perfect group for this kind of intervention, because we believe interns should start their careers with high regard for systematic quality investigation that integrates provider input.

Differences in Case-Patient Identification

Our second hypothesis was that the record review and the physician reporting methods would uncover the same set of adverse events. Although both physician reporting and chart-review methods found nearly the same number of adverse events, there was only moderate agreement between them. Of 133 events noted by the two methods, only 41 were identified in common. Because other evaluations of the quality of care have shown various patterns of socioeconomic risk for poor care [16, 17], we thought it was important to determine whether the two strategies were selectively identifying certain classes of patients with adverse events. But neither socioeconomic status, type of adverse events, nor severity of injury were different between the two strategies. Nor did our review of the clinical details of the patients' hospitalizations show important differences.

Certain factors do, however, explain a portion of the patients identified by the housestaff but not noted by the record review. Twenty-nine percent of those patients (14 of 48) were screened negative by medical-record analysts. This raises questions about the screening criteria used for this study. Whereas they are widely used by hospital quality-assurance teams and by peer-review organizations, our data suggest that their strengths and weakness need more exploration, a process that we have started to do (Bates DW and colleagues. Unpublished data). Among the remaining adverse events not detected in the retrospective record review, we found that some charts did not fully reflect the clinical scenario described by the house officers in the confidential physician reporting system. The rest were patients in whom the adverse event was apparent in the record but was not identified by reviewing physicians. Although we carefully designed our data collection instruments to focus reviewers' attention on critical variables contained in the medical record, our reviewers did not identify every adverse event and could not identify adverse events that providers did not detail in the medical record.

With regard to patients found by the record review but not by the housestaff, this was probably due to lack of participation by the housestaff. Because the physician report strategy was intended to operate without disruption of clinical care and because participation was completely voluntary, minimal effort was exerted to encourage reporting. Using electronic mail, daily prompts were sent by a project director who had little other contact with the housestaff. We worked closely with the leadership of the department to gain their approval, support, and advice about this effort but never sought to have them exercise authority over the housestaff. Hence, some lack of participation was expected. Yet only a few house officers (4 of 46) did not participate.

Cooperation from Housestaff

Our independence from the usual lines of authority in the hospital may have helped create a cooperative environment. Housestaff members could feel free to report to our study team because we had reassured them that we had no interest in sanctions against individuals but rather were interested in statistical information on risks for adverse events. In this regard, our effort is similar to that of the National Aeronautics and Space Administration (NASA) in general aviation. Pilots and controllers are encouraged to report errors to NASA, which then analyzes the information for trends to be reported back to the Federal Aviation Administration. Persons feel comfortable reporting information to NASA that they would not report to the Federal Aviation Administration [18]. With this phenomenon in mind, our hospital has now transformed our study team into a permanent mechanism for improving care through independent studies and interventions that revolve around provider participation.

Problems with Traditional Quality-Assurance Devices

Finally, the present study showed that our hospital identifies relatively few adverse events using existing quality-assurance mechanisms. We used, as a unit measure of quality, the occurrence of adverse events or injuries caused by medical management. This is only one component of quality. However, we found the quality-assurance devices at our hospital were generally aimed at identification of such iatrogenic injury. Moreover, we reviewed the quality-assurance records to identify those quality problems that met the definition of an adverse event. Quality problems such as patient dissatisfaction with communication were eliminated from the comparative analysis.

The patients identified by the conventional quality-assurance mechanisms were similar in their clinical context and severity to those found by the housestaff and the record review strategies. Some approaches, for instance those of the cardiac catheterization laboratory, were able to identify nearly as many of the quality problems as the combined methods. Others appeared to be less effective. Overall, existing quality-assurance mechanisms identified few adverse events compared with either a record review or a physician-reporting strategy. This raises important questions about the conventional methods for doing quality assurance in hospitals and emphasizes the need for innovation.

Acceptance and support by house officers for our method are perhaps best exemplified by the follow-up on this study. As part of this investigation, we did a case–control study to evaluate risk factors for adverse events. We identified several substantial factors and, working with housestaff, developed specific interventions. We are again in the process of reporting adverse events after implementation of the interventions. Throughout this process, the housestaff members have been supportive and eager to develop new ideas, especially about interventions. They have seen how data collection and analysis can lead to improvement in care and have none of the fear and dismay that often characterizes physician involvement in quality assurance. Indeed, their attitude is akin to what quality experts would call "kaizen" or gradual unending improvement [19].

We conclude that the physician reporting process is an identification strategy for uncovering quality problems that should be explored further. Although it does not indicate every quality problem, it does as well as a record review in identifying adverse events and is less expensive. It also actively integrates physicians into quality-improvement efforts. Of course, this investigation involved only one teaching hospital and one service and thus may not be generalizable. We plan to explore this issue using a multiple institution study of physician reporting, and we hope to show that interventions based on this kind of investigation can improve the quality of care.