Measuring and Improving Physician Compliance with Clinical Practice Guidelines: A Controlled Interventional Trial

Methods

Study Design

The study was a retrospective analysis of a prospective, controlled interventional trial [4].

Institutional Characteristics

The study was done at Cedars-Sinai Medical Center, a large university-affiliated community teaching hospital in Los Angeles. Most patients were cared for by private practitioners. Although a coronary care unit director, cardiology faculty, cardiology fellows, medical housestaff, and medical students participated in the clinical care of many of the study patients, private practitioners had the ultimate responsibility for patient care decisions for most study patients, including the decision to discharge a patient from the hospital.

Patient Population

The prospective, controlled interventional trial was done between 1 September 1991 and 31 August 1992. Physicians were informed of risk information and guideline recommendations on an alternating month basis. The retrospective analysis done for the current study included patients from all interventional months (October 1991, December 1991, February 1992, April 1992, June 1992, and August 1992). Patients were eligible for enrollment in the study if they were at least 30 years of age, presented with a chief symptom of chest pain, and were admitted to a monitored bed in the cardiac intensive care unit or the cardiac observation unit. Patients were enrolled in the study if they were concurrently classified as being at low risk for in-hospital complications according to our previously described practice guideline [4] or if they were identified through our hospital information system for retrospective review of all eligible patients admitted during intervention months. The guideline prescribed a 2-day length of hospital stay for patients classified as being at low risk. Noncompliance of physicians with the guideline was defined as a 3-day or greater length of stay. Analysis was by intention to treat.

Intervention

Before the study, all attending physicians received a letter describing the study and information about the chest pain practice guideline. The guideline [4] recommended a 2-day length of stay for patients hospitalized with chest pain who had none of the following conditions 24 hours after hospitalization: 1) acute myocardial infarction; 2) recurrent chest pain; 3) previous cardiac intervention; 4) planned cardiac intervention; 5) previous complication; 6) ongoing intervention; or 7) unstable comorbid condition. All seven elements describing low-risk status were explicitly defined. Physicians caring for patients classified as low-risk received concurrent, personalized, written, and physician-to-physician feedback over the telephone about the guideline recommendations. Both types of feedback were delivered to physicians using the following structured format:

Your patient (Patient Name) may be at low-risk for serious in-hospital complications. In fact, approximately 98% of patients similar to your patient suffered no serious in-hospital complications after 2 days of hospitalization. If your patient continues to have an uncomplicated hospital course without chest pain, please consider whether a 2 day length of stay is appropriate for your patient. If further diagnostic testing is absolutely required prior to discharge, please let me know if we can help facilitate any test for you. Only you as the primary physician can determine the most appropriate length of stay for your patient. Thank you.

No incentives were added or disincentives removed to enhance physician compliance with the guidelines. Specifically, no financial incentives were offered for complying with guideline recommendations, nor were any sanctions used against physicians who rejected the guidelines.

Inclusion Criteria

All patients eligible for enrollment in the study during intervention months who were classified at 24 hours as being at low risk by either concurrent utilization-management review or by retrospective review who were discharged 3 or more days after admission (our definition of noncompliance of physicians) were included in the analysis.

Sources of Information

Decisions about the possible cause or causes of noncompliance of physicians were based on review of the medical records and on the utilization-management physician-to-physician interaction sheets.

Analytic Framework

Patients were retrospectively assigned by an experienced research nurse to one of the following five general categories of physician noncompliance on the basis of a structured explicit review of medical records and of the utilization-management physician-to-physician interaction sheets (the combination was considered the criterion standard).

1. Noncompliance of Physicians Based on Guideline Implementation Problems

a) False-positive result: those patients misclassified as being at low risk during concurrent utilization-management review who after structured explicit review (the criterion standard) were considered high risk and, therefore, should not have been discharged according to the guideline's recommendation.

b) False-negative result: 1) those patients falsely classified as being at high risk and, therefore, the physician was not contacted; or 2) those patients not identified by utilization-management nurses as being at low risk but, instead, were identified retrospectively using the hospital information system and from retrospective structured explicit review (that is, patients not enrolled in the study despite meeting eligibility requirements).

2. Noncompliance of Physicians Based on Health Care System Inefficiencies

Patients correctly categorized as being at low risk whose physician agreed to discharge them but because of health care system inefficiency were not discharged before day 3. The delay tool [6] was used for the abstraction and categorization of inefficiency. Examples of delays included waiting for tests, procedures, or results; social issues (for example, waiting for nursing home placement); waiting for consultation; and patient refusal.

3. Noncompliance of Physicians Based on Changes in Patient Clinical Status between 24 and 48 Hours

Patients correctly classified at 24 hours as being at low risk but who between 24 and 48 hours either had a complication, an intervention planned (for example, cardiac catheterization), or an unstable comorbid condition. These events caused the patient to be reclassified as being at high risk on retrospective review; therefore, physician noncompliance was based on a changing medical condition rather than on physician, guideline, or hospital issues.

4. Physician Noncompliance Based on Physician Refusal To Discharge

This category was assigned if the following four conditions were met:

a) The patient was truly at low risk and did not change status between 24 and 48 hours.

b) Discharge was not delayed secondary to possible hospital inefficiencies.

c) The physician's refusal to discharge was documented on the utilization-management, physician-to-physician interaction sheet (these sheets were completed by utilization-management physicians after telephone contact with the attending physician)

d) The physician refused to discharge without giving a reason related to medical issues.

These patients' records were then independently reviewed by two internists blinded to the study hypothesis. They were instructed to answer the question: “Was there a medically appropriate reason to keep this patient hospitalized for further evaluation or treatment (or both) beyond 48 hours after admission?” Disagreement between the two physician reviewers was resolved by consensus.

5. Undetermined Reason for Noncompliance of Physicians

This category was composed of patients for whom no apparent reasons, as listed in the above four categories, could be found on retrospective review.

Patients in the “undetermined reason” group were compared with truly low-risk patients discharged according to the guideline using comorbidity scores [7] and severity of illness as measured by the acute cardiac ischemia time-insensitive predictive instrument scores [8]. Comorbidity scores begin at zero (indicating an absence of comorbid illnesses); the higher the score, the more comorbid illnesses are present [7]. Time-insensitive predictive instrument scores range from 0 to 100; the higher the score, the greater the probability for acute ischemia [8].

Statistical Analysis

A P value ≤ 0.05 was considered statistically significant. Categorical variables were compared using chi-square analysis. The statistical software packages GB-Stat and True Epistat were used [9, 10]. Where indicated, data are given as mean ±SD.

Data Quality

Of 1531 patients who were potentially eligible for inclusion in the prospective, controlled interventional trial, 5% were reviewed independently by two experienced research nurses for the inter-rater reliability of risk status. The κ value for these 82 randomly selected patients was 0.94. Ninety-nine percent of the 1531 patient records were available for review.

For the 46 truly low-risk patients who were discharged 3 or more days after admission, 100% had medical records that were available for review and 94.5% (43 patients) had utilization-management physician-to-physician interaction sheets containing usable information available for analysis. All 46 patients who were low risk during intervention months and who were discharged 3 or more days after admission were reviewed independently by two experienced research nurses to determine the inter-rater reliability for the reason for physician noncompliance; the κ value was 0.65.

Results

During intervention months, 230 patients were enrolled in the prospective, controlled interventional trial. Of these, 151 (66%) were discharged in 2 days or less. Figure 1 summarizes the reasons for physician noncompliance found on retrospective analysis for the remaining 79 (34%) patients who had a length of stay of 3 or more days. Of these 79 patients, 23 (29%) were misclassified during concurrent review as being at low risk, but these patients were determined on retrospective structured explicit review (the criterion standard) to be at high risk (false-positive result). None of these misclassified patients died or had a substantial complication on 30-day follow-up. Ten patients (13%) were initially misclassified as being at high risk when retrospective review showed that they were truly at low risk (false-negative result) or were never reviewed concurrently. Thus, of 79 patients whose discharge was not compliant with the guideline, 33 (42%) were discharged because of guideline misclassification.

Of the remaining 46 truly low-risk patients (true-positive result), four subgroups were delineated according to the reason for physician noncompliance. In 7 patients, clinical status changed between classification at 24 hours and potential hospital discharge. All these patients had a clinical event that caused their reclassification to high risk at 48 hours and made them inappropriate for discharge according to the guideline. These events included recurrent chest pain (2 patients), unstable atrial fibrillation (1 patient), planned cardiac catheterization (2 patients), and administration of intravenous antibiotics (2 patients). For 11 patients, although the physician agreed with discharge, a hospital or health system inefficiency prevented compliance with the guideline. For 8 of these 11 patients, this was because of test scheduling problems; of the remaining 3 patients, 1 had unavailable test results, 1 did not have a bed available at the nursing home, and 1 patient refused discharge. The retrospective review showed an undetermined cause for physician noncompliance in another group of 15 patients and for physician refusal to discharge without a stated medical reason in the remaining 13 patients.

Further analysis was done of these 15 patients who were accurately classified as being at low risk and who did not have an obvious reason for physician noncompliance or physician refusal to discharge. A significant difference was found in the Charlson comorbidity scores [6] between these patients and truly low-risk patients who were discharged according to the guideline. The mean comorbidity score (±SD) was 0.81 ±1.1 for patients of compliant physicians compared with 1.4 ±1.3 for patients of noncompliant physicians (P = 0.05). Comparison of the severity of illness as measured by the acute coronary ischemia time-insensitive predictive instrument [7] also showed significant differences. The mean time-insensitive predictive instrument scores (±SD) were 31.5 ±14.3 for patients of compliant physicians compared with 41.3 ±14.1 for patients of noncompliant physicians (P = 0.017).

Further review of the 13 patients who had a documented physician refusal to discharge according to the guideline showed no single physician responsible for more than 1 patient. Eight refusals (61.5%) were by cardiologists serving as either primary physicians or consultants. Three refusals (23%) were by trainees (either residents or fellows), and 2 refusals (15%) were by internist attendings. The distribution of the types of physician who refused to discharge their patients was not statistically different (P = 0.63) compared with physicians who discharged their truly low-risk patients according to the guideline (a similar distribution was noted for internists, cardiologists, and housestaff).

Structured implicit review of these 13 patient records in the physician refusal category showed no medically appropriate reason for hospitalization beyond day 2 in 7 of the patients, medically appropriate reasons not contained in the guideline in 5 patients, and inability to determine medical appropriateness in 1 patient. The two independent reviewers initially agreed on the risk status for 9 patients (κ, 0.28), and they finally agreed on risk status by consensus for the remaining 4 patients.

Discussion

Despite the current enthusiasm for clinical practice guidelines, few controlled interventional trials show their effectiveness and safety. In addition, important questions have recently been raised about the accuracy and appropriateness of guidelines (for example, sensitivity and specificity and positive and negative predictive values) for various medical interventions [11]. If guidelines are to be used to change physician practices and to assess physicians' willingness to participate in the effort to reduce health care costs, many important questions remain to be answered.

This retrospective analysis of physician compliance with a clinical practice guideline was part of a controlled interventional trial of a guideline designed to decrease hospital length of stay for selected low-risk patients admitted with chest pain [4]. The reason that 69% of physicians complied with the guideline in the original study and only 64% complied in this analysis is the inclusion of patients with false-positive results for this study (those patients misclassified as being at low risk at concurrent review but who were truly at high risk). By most criteria, this was a successfully implemented guideline. In the original study, the guideline improved physician compliance with the recommendation for discharge from 50% in control months to 69% in intervention months (P = 0.001); significantly decreased the total costs of care; and showed no adverse effect on patient morbidity, mortality, health status, or satisfaction as measured 30 days after discharge. Before analyzing one of the study's potential shortcomings (only 69% of physician's complied with the guideline recommendation), we should comment on why it appeared to be successful overall. First, the guideline recommendation was delivered as risk information to the attending physician. The risk information was based on outcome data from previous retrospective and prospective trials of the same guideline. Greco and Eisenberg [12] recently commented that “when cost reduction is the primary goal, physicians are likely to respond more favorably to evidence that the proposed changes improve patient outcomes or at least that patient outcomes are not worsened.”

Second, six general methods are available for changing physicians' practice behaviors. These include education, feedback, participation by physicians in efforts to bring about change, administrative rules, financial incentives, and financial penalties [12]. In general, combinations of methods are superior to single methods of intervention [12]. Although we did not use financial incentives, physicians had the financial disincentive of fewer billable patient visits in this predominantly Medicare population. In addition, we did not use administrative rules. However, we did combine education, physician participation in guideline development, and concurrent written and verbal feedback by utilization-management nurses and respected peer physicians.

Despite the overall success of the guidelines, our analysis of the reasons that physicians apparently did not comply with the guidelines points out and re-emphasizes several important issues in guideline development and implementation. The first is that guidelines are like any other diagnostic test in that they have a measurable sensitivity, specificity, and positive and negative predictive values. We found that during the first 5 intervention months of implementation by the utilization-management department, this guideline had a sensitivity of 0.85 and a specificity of 0.90. The positive and negative predictive values were 0.76 and 0.94, respectively. These observations raise the issues of safety and effectiveness of implementation at our institution and at other institutions [13]. However, in that study [13], no patient discharged according to guideline recommendations died either in the hospital or within 30 days of discharge. Thus, excellent patient outcomes were maintained.

These observations do have important implications when assessment of physician compliance with practice guidelines is attempted. If one is measuring physician performance (and perhaps even considering its use in making decisions about health plan participation or employment), the sensitivity and specificity of a given guideline or practice measure have profound implications. In our study, 42% of all apparent physician noncompliance was caused by guideline classification (implementation) errors. A false-positive classification (concurrent utilization-management review classifying the patient as being at low risk when they were truly at high risk) actually requires that the physician override the guideline; yet, by doing so, the physician may appear to be noncompliant. For false-negative classifications, the physician may never be contacted. One would not necessarily expect the physician to comply.

The next most common reason for apparent physician noncompliance was related to the patient's medical condition, including change in clinical status and increased severity-of-illness score. In 7 cases (9%), the patient had some event that changed her or his classification to high risk between categorization at 24 hours and recommended discharge at 48 hours. Again, it is the responsibility of the physician to review all guideline recommendations in light of recent events and changes in patient clinical status. This also emphasizes the importance of updating guideline recommendations during a patient's hospitalization.

Careful review of the 15 patients (19%) for whom we could identify no clear reason for physician noncompliance with the guideline recommendation to discharge showed statistically significant differences in comorbidity and severity of illness. No guideline can delineate all of the clinical subtleties of a patient. The fact that these patients were sicker and that their physicians appeared to be aware of this condition reinforces the importance of ongoing physician review and the need to use guidelines to complement rather than replace physician judgment. This also supports Tanenbaum's [14] recent reflection that “physicians should assert the legitimacy—indeed the necessity—of reasoning about individual patients on the basis of personal experience and theories of cause and effect, as well as on the basis of statistical knowledge.” This observation further supports the conclusions of Salem-Schatz and colleagues [15] that “failure to adjust for case mix in physician practice profiles may lead to overestimates of variations and misidentification of outliers.”

Next, the issue of inefficient hospitals and health care systems must be addressed. Health care system inefficiencies accounted for 14% of the apparent noncompliance of physicians in our study. Selker and colleagues [6] have shown that in one tertiary care teaching hospital, 30% of all patients had delays and that the average delay was 2.9 days because of hospital or system inefficiencies. In addition, the effect of weekends on hospital stay was recently evaluated [16] and was shown to have a substantial effect on length of stay in part because of physician coverage issues. Again, if the physician's compliance is used as the unit of analysis, the system measuring and attempting to encourage improved efficiency may be the cause of the problem.

Finally, for 13 patients (16%), physician refusal to discharge a patient appeared to be the reason for noncompliance. Our analysis did not show any one physician who was consistently noncompliant, but because our guideline reached many primary physicians, cardiologists, and trainees, it is difficult to detect a noncompliant pattern in a single physician. Nevertheless, our observation is consistent with current continuous quality improvement principles that, in general, state that most perceived problems are related to system inefficiencies and, less commonly, to individual issues [17]. In addition, the in-depth review of patients' records in this group showed complicated and challenging “real world” issues in decision making. The finding that two independent reviewers blinded to the study hypothesis eventually agreed that 5 of the 13 patients in the physician refusal group should not have been discharged is of interest. In addition, the lack of initial agreement between the two reviewers about appropriateness of continued hospitalization (κ, only 0.28) and the fact that in one patient no opinion could be given suggests that physician noncompliance with guideline recommendations was not based on rejection of the concept of guidelines but rather indicates a struggle with complex clinical issues. Thus, our estimate that outright physician refusal represents 16% of apparent noncompliance of physicians with our clinical practice guideline may substantially overestimate the extent of this problem.

Our study had important limitations. First, it was a retrospective evaluation of a prospective, controlled interventional trial. Although much of the important data was available, the subjective categorization by research nurses must be kept in mind. Second, these observations were for a length-of-stay guideline. Causes of apparent noncompliance of physicians with other types of guidelines, such as appropriateness of procedures or prevention recommendations, may be different.

Our findings have several important implications for implementing clinical practice guidelines, measuring their performance and physician compliance, and improving their effectiveness. First, clinical practice guidelines have measurable sensitivities and specificities when implemented. Second, measuring physician compliance has many important implications, and great care must be taken when assuming that the cause of physician noncompliance is related to physician resistance rather than to implementation problems, health system inefficiencies, true severity-of-illness differences, or a change in patient clinical status. Attempts to improve guideline performance need to address far more than physician buy-in and must also address implementation systems and the real clinical limitations of practice guidelines. Finally, because no test, guideline, or pathway will have 100% sensitivity and specificity, compliance should always be less than 100%. We have shown that patients change their clinical status, have subtle but potentially important increases in severity of illness, and can be truly high risk yet falsely classified by concurrent review as being at low risk. Absolute compliance with guidelines implies that physicians are blindly following recommendations and represents a potential threat to patient outcomes and quality of care.