A Three-Part Intervention To Change the Use of Hormone Replacement Therapy in Response to New Evidence

20 July 2004 | Volume 141 Issue 2 | Pages 118-125

Background: Slow adaptation of new information by providers may result in suboptimal care.

Objective: To evaluate changes in prescriptions for combination hormone replacement therapy (HRT) after a multicomponent intervention to deliver new information to patients and providers.

Design: Quasi-experimental study with multiple baselines.

Setting: Veterans Affairs Tennessee Valley Healthcare System (VA-TVHS).

Patients: Female veterans age 50 to 79 years who had a prescription filled at the VA-TVHS for combination HRT between 1 January 2002 and 1 July 2002.

Measurements: Discontinuation of HRT.

Intervention: A 3-part intervention consisted of 1) notifying patients who were using combination HRT of the results of the Women's Health Initiative study (patient education component), 2) sending all providers an e-mail with the Women's Health Initiative study results (provider education component), and 3) placing an electronic alert in each eligible patient's chart (provider care component). The alert asked providers to reevaluate the need for combination HRT. The intervention was implemented at different VA-TVHS sites in a stepwise fashion to differentiate intervention effect from media effect. Study follow-up continued through 31 December 2002.

Results: The total rate of discontinuation of combination HRT was 70.3% in 2002. The proportion of discontinuation from time of media release until intervention was 23.3%. After initiation of the intervention, an additional 43% of the original cohort discontinued use of HRT; this percentage represents a 59% relative decrease in HRT use among patients. After adjustment for time, the discontinuation rate per day was 4.9 times higher after the multifacted intervention than after the media release (95% CI, 1.8 to 13.1).

Limitations: A true control group is lacking.

Conclusion: A multifaceted approach in an integrated health care system with standardized methods of communication is an effective way to implement patient-centered, effective, and timely care with changing medical knowledge.

Editors' Notes Top Editors' Notes Methods Results Discussion Author & Article Info References Context Evidence documents that months to years often elapse before physicians adopt clinical practices supported by randomized, controlled trials. These researchers studied interventions to decrease this lag time after publication of the Women's Health Initiative, a trial that concluded that the overall risks of combination hormone replacement therapy exceeded its benefits. Contribution A multicomponent intervention (patient notification, provider education, electronic alert in patient chart) seemed to increase the discontinuation rate of hormone replacement therapy use after release of study results from the Women's Health Initiative in July 2002. Caution The study did not include a true control group. –The Editors

 

The Women's Health Initiative (WHI), a large National Institutes of Health–sponsored primary prevention trial of combination hormone replacement therapy (HRT), was stopped early because the overall risk outweighed the benefits (1). The risks included nonfatal myocardial infarction and coronary deaths, which combined were the study's primary end point. The results of other recent randomized, placebo-controlled trials have also called into question the benefits of HRT (1-8). On the basis of this evidence, several organizations have now changed their recommendations on the use of combination HRT (9-13). For example, in November 2002, the U.S. Preventive Health Services Task Force recommended against prescribing combination HRT to prevent disease (14, 15).

Changes in the recommendations for HRT use represent a small fraction of an escalating amount of information that providers must process and put into practice. New clinical knowledge, such as revised recommendations for HRT use, requires providers to change their behavior (16, 17). Consensus guidelines, continuing medical education, and reminders are common, albeit marginally effective, strategies used to influence behavior (18-27). Additional strategies, such as formulary changes, can be perceived as institutional control mechanisms and, therefore, may be resisted by providers (28, 29). Strategies for change often fail because they do not consider patient or provider preferences, provider time burden, delays introduced by waiting until the patient's next office visit, and perceptions that the change may stifle the patient–provider relationship (30-36).

We sought to address some of the gaps in the current array of interventions for integrating new clinical knowledge into daily practice by developing an approach that is timely, patient-centered, and effective (30, 37, 38). Our goal was to inform both patients and providers about the risks and benefits of HRT and to facilitate an interaction by which that information could be applied without interfering in the provider–patient relationship.

Methods

Top Editors' Notes Methods Results Discussion Author & Article Info References

Setting

The Veterans Health Administration nationwide is composed of 23 Veterans Integrated Service Networks. Each network comprises hospitals, ambulatory facilities, and community-based outpatient centers. In Veterans Integrated Service Network 9, located in the mid-south and including the states of Kentucky and Tennessee, there are more than 1 million veterans, including approximately 65 000 women. Almost half of these female veterans are older than 45 years of age (Alvarez V. Personal communication). Of the 6 health care systems in the Veterans Integrated Service Network 9, the Veterans Affairs Tennessee Valley Healthcare System (VA-TVHS), a convenience sample, was selected to be the intervention group. The VA-TVHS comprises 2 teaching hospitals and 8 community-based outpatient centers. The VA-TVHS offers outpatient primary, secondary, and tertiary care to more than 150 000 veterans who are predominantly in middle Tennessee. Ambulatory care, including women's health care, is provided at all of the sites. In 2002, 2576 female veterans older than 45 years of age enrolled for care within the VA-TVHS (Alvarez V. Personal communication).

Study Design

This quality improvement project used a prospective, quasi-experimental intervention design. All female veterans using combination HRT and their providers were notified of the WHI study results in a staggered fashion. The study intervention was implemented at 3 sites; start times were separated by 2 weeks (Figure 1).

View larger version (13K): [in this window] [in a new window]   Figure 1. Study timeline. ACY = Alvin C. York Medical Center; CBOC = community-based outpatient center.

 

Given the media coverage surrounding the WHI study, this design was chosen to account for the secular trends that may have affected rates of HRT discontinuation (39). The WHI study results appeared in the 17 July 2002 issue of the Journal of the American Medical Association but were released to the press 1 week earlier. The WHI study results also appeared in the 10 July 2002 edition of The New York Times and other national and local papers.

The Nashville Veterans Affairs Medical Center was the first site for intervention, which began on 1 September 2002. At that point, the other 2 sites, which were exposed only to the media information on HRT, continued current care. The study intervention was replicated for all patients and providers at the community-based outpatient centers on 16 September 2002 and at the Alvin C. York Veterans Affairs Medical Center on 30 September and 1 October 2002. This staggered design was implemented to allow for concurrent comparison with usual care until the final group of the cohort underwent the intervention.

Patient Identification

We identified female veterans with a prescription filled for combination HRT by using the VA-TVHS pharmacy database. The pharmacy collects prescribing information in a relational database. This information is downloaded each month from the Veterans Affairs mainframe computer by a pharmacy supervisor. Search terms were Prempro (Wyeth-Ayerst, Philadelphia, Pennsylvania), Premphase (Wyeth-Ayerst), and medroxyprogesterone linked to any strength conjugated estrogens (0.3 mg, 0.625 mg, 0.9 mg, 1.25 mg, or 2.5 mg). We initially identified 131 veterans with this strategy. Using the WHI inclusion criteria, we limited our sample to female veterans age 50 to 79 years. To establish a clear baseline, we limited the sample to patients who had a prescription filled between 1 January 2002 and 1 July 2002. The final cohort consisted of 91 female veterans.

Intervention

Before implementing the intervention, leadership within the quality improvement, pharmacy, and primary care departments discussed and agreed on the design of the intervention. The intervention had 3 components: Two focused on education (targeting patient and provider) and 1 integrated new information with patient care. The provider education component ensured that all of the prescribing providers received information about the WHI study (Appendix Figure 1). This involved sending an e-mail message from the chief of pharmacy services and the medical director of the Pharmacy and Therapeutics Committee to all clinical chiefs explaining the planned intervention. Each clinical chief disseminated the e-mail to practitioners within their division. The e-mail also included an electronic link to the WHI study (http://jama.ama-assn.org/cgi/reprint/288/3/321.pdf) (1).

View larger version (30K): [in this window] [in a new window]   Appendix Figure 1. Provider education e-mail.

 
The patient education component of the intervention involved sending a personalized letter from the TVHS pharmacy to current users of combination HRT. This letter was signed by the medical director of the Pharmacy and Therapeutics Committee (Appendix Figure 2). The letter reported that WHI study participants taking combination HRT had higher risks for disease, including "a slight increase in the chance of developing breast cancer, a trend toward more heart attacks and strokes, and more likely to have blood clots in their veins." The letter acknowledged benefits of combination HRT for relief of menopausal symptoms in many women and emphasized that, "We recommend that patients and providers review the risks and benefits of HRT therapy on a personal basis."

View larger version (35K): [in this window] [in a new window]   Appendix Figure 2. Patient education letter.

 
A "pharmacy alert" was placed in the electronic chart of each patient identified to be using combination HRT in order to facilitate a timely conversation between the patient and the provider. This component served to overcome a common barrier to action: the need for the provider to identify all of the patients affected by new information. The pharmacy alert was an electronic notification that was sent, by the pharmacy, to the prescribing provider via each patient's electronic medical record. It contained a summary of the WHI study results and provided an electronic link to the WHI study as a reinforcement of the educational component. The Veterans Affairs network uses a computerized patient record system that allows for real-time processing of information. Each time that providers would sign on to a computer, any medical record that contained an electronic alert would be brought to their attention. The pharmacy alert asked the provider to reevaluate the need for combination HRT and generate an addendum to this alert. The addendum gave the provider several options for action, including continuation or discontinuation of combination HRT at the current time or at some point in the future (Appendix Figure 3).

View larger version (31K): [in this window] [in a new window]   Appendix Figure 3. Pharmacy alert.

 
Measurement

The primary outcome measure was the percentage of female veterans using combination HRT at the beginning of the study who discontinued this therapy. The measure of discontinuation was assessed through 2 steps. First, if the primary provider, in response to the chart alert, recorded the discontinuation order, then the date that the order was entered could be ascertained by reviewing the addendum to the pharmacy alert. When an addendum to the original note is generated, the results of that addendum automatically return to the original author of the pharmacy alert, allowing for collection of responses. Second, we reviewed charts at study completion to confirm whether combination HRT was discontinued. We also confirmed the date the discontinuation order was placed in the patient's electronic medical record.

Upon completion of the study, we repeated the initial search strategy in the pharmacy database to ascertain whether patients who had discontinued combination HRT reinitiated this therapy. The covariates of race and Charlson comorbidity score (40) were obtained through baseline chart review. One investigator reviewed the charts and calculated comorbidity scores from active problem lists kept in the patient's electronic medical record. Race was determined through patient self-identified demographic information. Covariates including age, sex, prescribing provider's division, and type of provider (staff physician, resident, or nurse practitioner) were collected through the pharmacy database.

Statistical Analysis

We calculated discontinuation rates for baseline, media release, and intervention periods by using as a denominator the number of people at risk for discontinuation at that time period. For example, the discontinuation rate shown was used to evaluate the overall trend for discontinuation contributed by the intervention.

We conducted an exploratory analysis by using the Student t-test and chi-square test to examine the effect of comorbid disease, demographic characteristics, and provider type on the outcome of discontinuation. All de-identified data within the data set were analyzed by using Cox proportional hazards with discrete time-varying covariates. This model assessed the association among explanatory variables (including the chart alert, Charlson comorbidity score, demographic variables, and provider type) with the outcome of discontinuation while adjusting for time. The model adjusted for the staggered implementation of the intervention and maintained a concurrent comparator group until the final group underwent the intervention. We constructed Kaplan–Meier curves for each intervention group and the media period to illustrate the rates of HRT discontinuation. All analyses were conducted by using Stata software, version 7 (Stata Corp., College Station, Texas). We obtained approval from the institutional review board of the primary institution for analysis of de-identified data.

Role of the Funding Source

The funding agency had no role in the design, conduct, or analysis of the data or in the decision to submit the manuscript for publication.

Results

Top Editors' Notes Methods Results Discussion Author & Article Info References

Patient Characteristics

The mean age of female veterans in the initial TVHS cohort (Table 1) was 57 years. The cohort was primarily white (94.4%) and relatively healthy. Most of the cohort (48 of 91 [53%]) had a Charlson comorbidity score of 0. This finding reflects a sample that is free of diabetes, heart disease, malignant conditions, chronic pulmonary disease, connective tissue disease, cerebrovascular disease, peripheral vascular disease, renal disease, and liver disease. More than 80% of the patients received care in 1 of the 2 hospital-based clinics. The remaining 20% of the cohort was seen in 1 of the 8 community-based outpatient centers. A midlevel provider (nurse practitioner or physician assistant) cared for 42% of the patients; the remainder received care from a staff physician or resident physician. Patient characteristics (age, Charlson comorbidity score, race, and provider) did not differ between women who discontinued HRT and those who continued the therapy (Table 1).

View this table: [in this window] [in a new window]   Table 1. Characteristics of the Tennessee Valley Healthcare System Cohort

 

Responder and Response Characteristics

We received 69 responses (75.8%) to 91 alerts placed in patients' charts. Twelve nurse practitioners and physician assistants responded to the alert for 34 of 39 (87.2% [95% CI, 77% to 91%]) of their patients, 18 staff physicians responded for 27 of 36 (75% [CI, 67% to 85%]) of their patients, and 13 resident physicians responded for 8 of 16 (50% [CI, 36% to 66%]) of their patients. Providers selected option 1 (spoke to the patient who wishes to continue using combination HRT [Appendix Figure 3]) for 6 of the 69 responses (9%). Providers selected option 2 (spoke to the patient who wishes to discontinue combination HRT [Appendix Figure 3]) for 31 of the 69 responses (44.8%). Providers selected option 3 (continue combination HRT and speak to the patient later [Appendix Figure 3]) for 21 of the 69 responses (32%). Providers selected option 4 (discontinue combination HRT now and speak to the patient later [Appendix Figure 3]) for 4 of the 69 responses (5.8%). For the remaining 6 responses (9%), providers selected option 5 (the patient is not in their practice [Appendix Figure 3]), and the alert was then forwarded to the appropriate provider (4 patients). Of the 4 forwarded alerts, providers reported that 1 patient chose to continue HRT and 3 discontinued HRT. An appropriate provider could not be identified for the remaining 2 patients.

Discontinuation Rates in the VA-TVHS

The total rate of discontinuation for 2002 in the VA-TVHS was 70.3% (64 of 91 patients). Five of the 91 patients (5.5%) discontinued using combination HRT between 1 January and 9 July 2002, before the WHI media release. After the media release and before the initiation of our multifactorial intervention, 20 additional patients discontinued use of combination HRT; they represented 22% of the original 91 patients and 23% of the 86 who continued to use HRT. After initiation of the intervention, an additional 39 patients discontinued use of combination HRT; they represented 43% of the original 91 patients and 59% of the 66 who continued to use HRT at the time of the intervention (Table 2).

View this table: [in this window] [in a new window]   Table 2. Discontinuation of Hormone Replacement Therapy by Site of Care over Time

 

The cumulative probability of discontinuing HRT was 2.57 times greater (59% vs. 23%) after the intervention than after the media release, which corresponds to an additional 43% of the cohort. These probabilities, however, represent 3 variable time periods. The baseline period was 171 days, the media period ranged from 54 to 83 days, and the intervention period ranged from time zero (intervention) to 134 days after the intervention was instituted. We used survival analysis (Cox proportional hazards model with discrete time-varying covariates) and Kaplan–Meier estimates to account for the time variation that each patient spent as a control. These analyses showed that the discontinuation rate per day was 4.9 times higher after the intervention than it was after the media release (CI, 1.8 to 13.1 [P = 0.002]) (Figure 2). A multivariate model that simultaneously adjusted for age, race, Charlson comorbidity score, and type of provider yielded results similar to the unadjusted model.

View larger version (19K): [in this window] [in a new window]   Figure 2. Discontinuation of hormone replacement therapy (HRT). Hazard ratio, 4.9 (95% CI, 1.8 to 13.1). ACY = Alvin C. York Medical Center; CBOC = community-based outpatient center.

 

At the completion of the study, the HRT drug query was repeated in the VA-TVHS pharmacy database. No new prescriptions for HRT were initiated after 10 July 2002, the media release date. None of the patients in the cohort who had discontinued combination HRT reinitiated the therapy.

Discussion

Top Editors' Notes Methods Results Discussion Author & Article Info References

Our providers were 4.9 times more likely to discontinue combination HRT use after the institution of the multifaceted intervention than after release of the WHI study results to the media. The intervention consisted of a 3-faceted approach that was implemented without an attempt to discern the relative contribution of each element. Changes in organizational structures that are directed at several layers within the system and that promote communication may be the most effective in producing desired results. Davis and colleagues have noted that multifaceted interventions are more effective than single interventions (19). By establishing a 2-way communication pathway between opinion leaders, frontline providers, and patients, knowledge can be efficiently transferred to those who are most likely to implement and benefit from it (41, 42). This communication enhances patient-centered and timely care, thereby fulfilling 2 of the 6 Institute of Medicine domains of health care quality (43).

Our results are comparable to those of Bero and colleagues, Revere and colleagues, and Oxman and colleagues (18, 44, 45), who conducted systematic reviews of computerized decision support systems. Their results demonstrated that computerized support leads to improved physician performance in selecting drug dosages, providing preventive care, and managing patients in general (46). Although computerized reminders have effectively improved patient care (47-49), they work only when integrated into the existing system. In addition, they remain highly dependent on an office-based interaction with the patient.

Institutional control mechanisms are often seen as diminishing patient and provider autonomy (27, 29). We attempted to empower providers to address changes in a patient's medication without waiting for the patient to come to the office. Often, medications and changes can be discussed during a telephone conversation (50). This approach can allow busy providers, who want to maintain the provider–patient relationship, to share new medical knowledge with their patients and make necessary adjustments.

The response to the alert differed by provider type. Resident physicians had the lowest response rates. Response rates did not differ significantly between midlevel providers and staff physicians. When responses from the residents were compared with those of either midlevel providers or staff physicians alone, the difference was significant (P < 0.002). Care delivered by residents may be fragmented because many residents typically conduct primary care clinic 1 afternoon a week. Many studies have compared resident to staff physicians regarding patient counseling and behavioral change interventions (51-57). Resident physicians lagged behind physicians in their ability to counsel patients effectively (53, 55). Resident education is a setting that requires feedback from supervisors, especially as it relates to patient care and counseling (58, 59). The investigators did not explore why knowledge integration differed at various levels of training.

Limitations of a time series study include confounding by secular trends and learning effects. We recognized that a secular trend, that is, the media effect, would influence the information that patients would obtain and consequently affect the rate of discontinuation of combination HRT. We minimized secular trends as an alternative explanation by using a study design of staggered intervention and analysis with time-dependent covariates. Familiarity with the alert by a few providers who practice at more than 1 site within the VA-TVHS may have contributed to increased discontinuation of combination HRT among patients who had not yet received the intervention. This contamination effect, however, would have biased our results to the null. Although the study design had multiple baselines, which accounted for secular trends, a true control group for comparison would have better determined the effect of the intervention. In addition, anecdotal feedback about provider satisfaction with this intervention has been positive; however, we did not systematically assess provider or patient satisfaction. Finally, generalizability of our study is limited by our use of organizational structures that have computerized patient record systems and electronic alert messaging systems.

Although we did not undertake a formal cost analysis, we did assess the resources used to conduct the intervention. The intervention required time for implementation by 2 persons, the outpatient pharmacy supervisor and a physician trained in quality improvement. The outpatient pharmacy supervisor spent 10 hours performing searches in the relational database and using the mail merge to link patient names and addresses into the personalized patient education letter. The physician trained in quality improvement spent approximately 20 hours designing the patient and provider education letters and the chart alert, placing alerts in patient charts, and collecting responses. Given the stepwise deployment of the study design, these 20 hours spanned a 12-week period.

A systems-based approach to the complex problems associated with the changing evidence about medications can help eliminate delays in patient care and disseminate new clinical knowledge in an effective and timely manner (60, 61). A common challenge to health care systems is to develop strategies to inform and educate rather than enforce medical decisions (30). Telling the provider what to do is unlikely to achieve learning or improve the quality of care provided to the patient (62). Most successful strategies reduce complexity for the provider and optimize information processing (63-65). To integrate knowledge into practice, the practitioner must move from awareness of the new guidelines to agreement, adoption, and adherence to new clinical information (66).

Using real-time care instead of waiting for an office-based interaction, our patients and providers are notified of the expanding body of medical knowledge in a timely fashion. Through our intervention, we have moved beyond information delivery to implementation of improved medication prescribing. We have changed behaviors associated with combination HRT and empowered providers and patients to act upon the most current medical information in a timely fashion. This approach illustrates a successful response to a common challenge (30) and may serve as a model for responding to new knowledge in other health care systems.

Author and Article Information

Top Editors' Notes Methods Results Discussion Author & Article Info References

From Veterans Administration Tennessee Valley Healthcare System and Vanderbilt University, Nashville, Tennessee.

Grant Support: Drs. Roumie and Grogan were supported by the Office of Academic Affiliations, Department of Veterans Affairs, and the National Quality Scholars Program. Dr. Dittus received support from the Geriatrics Research and Education Clinical Center and the Tennessee Valley Health Services Research Center. Drs. Speroff and Elasy received support from the Veterans Affairs Targeted Research Enhancement Program. The resources and facilities at the Veterans Affairs Tennessee Valley Healthcare System, Nashville, Tennessee, were used to support this paper.

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: Christianne L. Roumie, MD, Nashville Veterans Affairs Medical Center, Geriatrics Research and Education Clinical Center 4A120, 1310 24th Avenue South, Nashville, TN 37212; e-mail, christianne.roumie{at}vanderbilt.edu.

Current Author Addresses: Drs. Roumie, Grogan, Speroff, Dittus, and Elasy: Vanderbilt University, Departments of Medicine and Surgery, Nashville Veterans Affairs Medical Center, Geriatrics Research and Education Clinical Center, 1310 24th Avenue South, Nashville, TN 37212.

Drs. Fable and Awad: Nashville Veterans Affairs Medical Center, Pharmacy Services, 1310 24th Avenue South, Nashville, TN 37212.

References

Top Editors' Notes Methods Results Discussion Author & Article Info References

1. Rossouw JE, Anderson GL, Prentice RL, LaCroix AZ, Kooperberg C, Stefanick ML, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. JAMA. 2002;288:321-33. [PMID: 12117397].[Abstract/Free Full Text]

2. Hulley S, Grady D, Bush T, Furberg C, Herrington D, Riggs B, et al. Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women. Heart and Estrogen/progestin Replacement Study (HERS) Research Group. JAMA. 1998;280:605-13. [PMID: 9718051].[Abstract/Free Full Text]

3. Rapp SR, Espeland MA, Shumaker SA, Henderson VW, Brunner RL, Manson JE, et al. Effect of estrogen plus progestin on global cognitive function in postmenopausal women: the Women's Health Initiative Memory Study: a randomized controlled trial. JAMA. 2003;289:2663-72. [PMID: 12771113].[Abstract/Free Full Text]

4. Wassertheil-Smoller S, Hendrix SL, Limacher M, Heiss G, Kooperberg C, Baird A, et al. Effect of estrogen plus progestin on stroke in postmenopausal women: the Women's Health Initiative: a randomized trial. JAMA. 2003;289:2673-84. [PMID: 12771114].[Abstract/Free Full Text]

5. Smith MA, Perry G, Atwood CS, Bowen RL. Estrogen replacement and risk of Alzheimer disease [Letter]. JAMA. 2003;289:1101-2. [PMID: 12622570].[Free Full Text]

6. Ylikorkala O, Metsa-Heikkila M. Hormone replacement therapy in women with a history of breast cancer. Gynecol Endocrinol. 2002;16:469-78. [PMID: 12626034].[Medline]

7. Hlatky MA, Boothroyd D, Vittinghoff E, Sharp P, Whooley MA. Quality-of-life and depressive symptoms in postmenopausal women after receiving hormone therapy: results from the Heart and Estrogen/Progestin Replacement Study (HERS) trial. JAMA. 2002;287:591-7. [PMID: 11829697].[Abstract/Free Full Text]

8. Hays J, Ockene JK, Brunner RL, Kotchen JM, Manson JE, Patterson RE, et al. Effects of estrogen plus progestin on health-related quality of life. N Engl J Med. 2003;348:1839-54. [PMID: 12642637].[Abstract/Free Full Text]

9. Postmenopausal hormone therapy: cardiovascular risks. Prescrire Int. 2003;12:65-9. [PMID: 12674130].[Medline]

10. Canadian consensus on hormone replacement therapy. Estrogen and progestin use in postmenopausal women. Can Fam Physician. 2003;49:188-91. [PMID: 12619742].[Free Full Text]

11. Carney PA, Miglioretti DL, Yankaskas BC, Kerlikowske K, Rosenberg R, Rutter CM, et al. Individual and combined effects of age, breast density, and hormone replacement therapy use on the accuracy of screening mammography. Ann Intern Med. 2003;138:168-75. [PMID: 12558355].[Abstract/Free Full Text]

12. Amended report from the NAMS Advisory Panel on Postmenopausal Hormone Therapy. Menopause. 2003;10:6-12. [PMID: 12544671].[Medline]

13. Role of progestogen in hormone therapy for postmenopausal women: position statement of The North American Menopause Society. Menopause. 2003;10:113-32. [PMID: 12627037].[Medline]

14. Postmenopausal hormone replacement therapy for primary prevention of chronic conditions: recommendations and rationale. Ann Intern Med. 2002;137:834-9. [PMID: 12435221].[Abstract/Free Full Text]

15. Kolata G. Study Is Halted Over Rise Seen in Cancer Risk. New York Times. 10 July 2002.

16. Berwick DM, Leape LL. Reducing errors in medicine [Editorial]. Qual Health Care. 1999;8:145-6. [PMID: 10847867].[Medline]

17. Leape LL. Institute of Medicine medical error figures are not exaggerated. JAMA. 2000;284:95-7. [PMID: 10872022].[Free Full Text]

18. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ. 1998;317:465-8. [PMID: 9703533].[Free Full Text]

19. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA. 1995;274:700-5. [PMID: 7650822].[Abstract]

20. Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282:867-74. [PMID: 10478694].[Abstract/Free Full Text]

21. Davis D. Does CME work? An analysis of the effect of educational activities on physician performance or health care outcomes. Int J Psychiatry Med. 1998;28:21-39. [PMID: 9617647].[Medline]

22. Davis D. Physician education, evidence and the coming of age of CME [Editorial]. J Gen Intern Med. 1996;11:705-6. [PMID: 9120659].[Medline]

23. Haynes RB, Davis DA, McKibbon A, Tugwell P. A critical appraisal of the efficacy of continuing medical education. JAMA. 1984;251:61-4. [PMID: 6690767].[Abstract]

24. Davis DA, Mazmanian PE. On CME and the reform of medical education. Acad Med. 1994;69:207-8. [PMID: 7695671].[Medline]

25. Davis D, Parboosingh J. "Academic" CME and the social contract. Acad Med. 1993;68:329-32. [PMID: 8484835].[Medline]

26. Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME. A review of 50 randomized controlled trials. JAMA. 1992;268:1111-7. [PMID: 1501333].[Abstract]

27. Tu K, Davis D. Can we alter physician behavior by educational methods? Lessons learned from studies of the management and follow-up of hypertension. J Contin Educ Health Prof. 2002;22:11-22. [PMID: 12004636].[Medline]

28. Runciman WB, Merry AF, Tito F. Error, blame, and the law in health care—an antipodean perspective. Ann Intern Med. 2003;138:974-9. [PMID: 12809454].[Abstract/Free Full Text]

29. Smalley W, Shatin D, Wysowski DK, Gurwitz J, Andrade SE, Goodman M, et al. Contraindicated use of cisapride: impact of food and drug administration regulatory action. JAMA. 2000;284:3036-9. [PMID: 11122591].[Abstract/Free Full Text]

30. Kassirer JP. The quality of care and the quality of measuring it [Editorial]. N Engl J Med. 1993;329:1263-5. [PMID: 8413394].[Free Full Text]

31. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8:527-34. [PMID: 11687560].[Abstract/Free Full Text]

32. Sim I, Owens DK, Lavori PW, Rennels GD. Electronic trial banks: a complementary method for reporting randomized trials. Med Decis Making. 2000;20:440-50. [PMID: 11059477].[Abstract/Free Full Text]

33. Sim I, Rennels G. A trial bank model for the publication of clinical trials. Proc Annu Symp Comput Appl Med Care. 1995;:863-7. [PMID: 8563415].

34. Eddy DM. Clinical decision making: from theory to practice. Three battles to watch in the 1990s. JAMA. 1993;270:520-6. [PMID: 8320794].[Medline]

35. Eddy DM. Clinical decision making: from theory to practice. Principles for making difficult decisions in difficult times. JAMA. 1994;271:1792-8. [PMID: 8196126].[Medline]

36. Eddy DM. Clinical decision making: from theory to practice. Benefit language: criteria that will improve quality while reducing costs. JAMA. 1996;275:650-7. [PMID: 8594249].[Medline]

37. Cash DW, Clark WC, Alcock F, Dickson NM, Eckley N, Guston DH, et al. Knowledge systems for sustainable development. Proc Natl Acad Sci U S A. 2003;100:8086-91. [PMID: 12777623].[Abstract/Free Full Text]

38. Davis D, Evans M, Jadad A, Perrier L, Rath D, Ryan D, et al. The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 2003;327:33-5. [PMID: 12842955].[Free Full Text]

39. Speroff T, O'Connor GT. Study designs for PDSA quality improvement research. Qual Manag Health Care. 2004;13:17-32. [PMID: 14976904].[Medline]

40. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40:373-83. [PMID: 3558716].[Medline]

41. Knowledge translation. Accessed at http://www.healthinfonet.ecu.edu.au/html/html_home/home_kt.htm on 17 May 2004.

42. Thomson O'Brien MA, Oxman AD, Haynes RB, Davis DA, Freemantle N, Harvey EL. Local opinion leaders (Systematic review). Cochrane Effective Practice and Organisation of Care Group. Cochrane Database of Systematic Reviews. 2003;1.

43. Chassin MR, Galvin RW. The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. JAMA. 1998;280:1000-5. [PMID: 9749483].[Abstract/Free Full Text]

44. Revere D, Dunbar PJ. Review of computer-generated outpatient health behavior interventions: clinical encounters "in absentia". J Am Med Inform Assoc. 2001;8:62-79. [PMID: 11141513].[Abstract/Free Full Text]

45. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ. 1995;153:1423-31. [PMID: 7585368].[Abstract]

46. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994;120:135-42. [PMID: 8256973].[Abstract/Free Full Text]

47. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med. 1976;295:1351-5. [PMID: 988482].[Abstract]

48. Bates DW. Using information technology to reduce rates of medication errors in hospitals. BMJ. 2000;320:788-91. [PMID: 10720369].[Free Full Text]

49. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6:313-21. [PMID: 10428004].[Abstract/Free Full Text]

50. Friedman RH. Automated telephone conversations to assess health behavior and deliver behavioral interventions. J Med Syst. 1998;22:95-102. [PMID: 9571516].[Medline]

51. Rodney WM, Beaber RJ, Johnson R, Quan M. Physician compliance with colorectal cancer screening (1978-1983): the impact of flexible sigmoidoscopy. J Fam Pract. 1985;20:265-9. [PMID: 3973543].[Medline]

52. Shapiro J. Does a behavioral science rotation change residents' self-assessed psychosocial attitudes and behavior? Fam Pract Res J. 1991;11:107-19. [PMID: 2028810].[Medline]

53. Kelly RB, Velez-Holvino O, Alemagno SA. Serum cholesterol: attitudes and behavior of family practice residents. J Fam Pract. 1991;33:259-65. [PMID: 1880484].[Medline]

54. McIlvain HE, Susman JL, Manners MA, Davis CM, Gilbert CS. Improving smoking cessation counseling by family practice residents. J Fam Pract. 1992;34:745-9. [PMID: 1593249].[Medline]

55. Evans AT, Rogers LQ, Peden JG Jr, Seelig CB, Layne RD, Levine MA, et al. Teaching dietary counseling skills to residents: patient and physician outcomes. The CADRE Study Group. Am J Prev Med. 1996;12:259-65. [PMID: 8874689].[Medline]

56. Goebel LJ. A peer review feedback method of promoting compliance with preventive care guidelines in a resident ambulatory care clinic. Jt Comm J Qual Improv. 1997;23:196-202. [PMID: 9142611].[Medline]

57. Nalven LM, Hofkosh D, Feldman H, Kelleher K. Teaching pediatric residents about early intervention and special education. J Dev Behav Pediatr. 1997;18:371-6. [PMID: 9431498].[Medline]

58. Denton GD, Smith J, Faust J, Holmboe E. Comparing the efficacy of staff versus housestaff instruction in an intervention to improve hypertension management. Acad Med. 2001;76:1257-60. [PMID: 11739054].[Medline]

59. Saitz R, Horton NJ, Sullivan LM, Moskowitz MA, Samet JH. Addressing alcohol problems in primary care: a cluster randomized, controlled trial of a systems intervention. The screening and intervention in primary care (SIP) study. Ann Intern Med. 2003;138:372-82. [PMID: 12614089].[Abstract/Free Full Text]

60. Berwick DM. Eleven worthy aims for clinical leadership of health system reform. JAMA. 1994;272:797-802. [PMID: 8078145].[Abstract]

61. Schlesinger LA, Heskett JL. The service-driven service company. Harv Bus Rev. 1991;69:71-81. [PMID: 10113913].[Medline]

62. Thomson O'Brien MA, Freemantle N, Oxman AD, Wolf F, Davis DA, Herrin J. Continuing education meetings and workshops (Systematic review). Cochrane Effective Practice and Organisation of Care Group. Cochrane Database of Systematic Reviews. 2003;1.

63. Leape LL. Preventing adverse drug events. Am J Health Syst Pharm. 1995;52:379-82. [PMID: 7757863].

64. Leape LL, Bates DW, Cullen DJ, Cooper J, Demonaco HJ, Gallivan T, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA. 1995;274:35-43. [PMID: 7791256].[Abstract]

65. Steinberg EP. Improving the quality of care—can we practice what we preach? [Editorial]. N Engl J Med. 2003;348:2681-3. [PMID: 12826644].[Free Full Text]