Using Systematic Reviews in Clinical Education

How Are Systematic Reviews Useful in Medical Training?

Traditional methods for keeping up with the burgeoning medical literature unfortunately do not increase the use of the literature by trainees [6-8]. For many years, we have urged physicians in training (and in practice) to read original research. We have used journal clubs to teach critical appraisal and to model how we as clinician-educators keep up with research. However, the impact of journal clubs on resident behavior has been disappointing [6-8]. Furthermore, neither trainees [9] nor faculty [10] want the faculty to attend journal clubs, and this reduces the opportunity to model efficient and effective use of the medical literature. Sadly, mandatory attendance and the provision of lunch are better than educational attributes at predicting which journal clubs will be well attended and will endure [9].

Many physicians in practice [11, 12] and in training [7] prefer to read traditional narrative review articles rather than original research studies. Previous authors in this series [13] and others [14-17] have discussed the scientific advantages that systematic reviews have over traditional reviews. We limit our discussion here to the characteristics of systematic reviews that are pertinent to medical education.

Systematic reviews can enhance training in several ways beyond providing trainees with reliable summaries of medical knowledge. First, a concise source of relevant evidence, such as that found in a systematic review, can encourage learning; in contrast, the time and effort involved in reading multiple original studies may discourage learning. Second, systematic reviews can help locate original studies when the trainee is not proficient in the electronic searching of the literature [18] or when the studies have not been published [19]. Similarly, systematic reviews can serve as an interface between original research and trainees who may be inexperienced at critical appraisal. This can mitigate, in part, concerns about trainees reading secondary rather than primary research publications. Third, exposing trainees to systematic reviews may decrease the knowledge gap between trainees and their teachers, increasing the trainees' confidence and fostering active learning behavior. If trainees share systematic reviews with their teachers, mutual learning may occur because faculty knowledge is often less extensive outside of specific areas of expertise. Finally, the Residency Review Commission for internal medicine [20] strongly encourages residents to participate in scholarly activity during training. The Commission and many residency program directors [20] consider the preparation of “analytic” reviews to be scholarly activity.

Limitations of Systematic Reviews as Educational Resources

It is late 1995, and a patient presents with severe alcoholic hepatitis, encephalopathy, and no gastrointestinal bleeding. You advise your ward team to use steroids because a recent meta-analysis [21] and a subsequent well-executed trial [22] both conclude that steroids reduce mortality in such patients. The team uses steroids on the basis of your suggestion, but the patient dies 5 days later of septic shock. A few months later, your resident shows you the current issue of ACP Journal Club[23], which contains an abstract from a more recent meta-analysis concluding that steroids do not help patients with alcoholic hepatitis. You worry not only that the steroids may have contributed to the patient's death from sepsis [1] but may not even have been indicated for alcoholic hepatitis [23]. This meta-analysis had been published before the patient died, and now you wonder whether you misinformed your ward team.

This scenario illustrates some of the limitations of systematic reviews. Trainees need to understand these limitations to use systematic reviews appropriately. Methodologic errors, and even bias, occur in systematic reviews just as they do in traditional reviews [24-26]. Different systematic reviews on the same topic may produce conflicting conclusions [27, 28], as might be expected with any new and rapidly evolving scientific method. Residents must therefore learn to assess the quality of systematic reviews in order to understand why they may conflict with each other [27, 28] or with the results of individual randomized trials [29].

Ways To Incorporate Systematic Reviews into Medical Training

The most obvious way to incorporate systematic reviews into clinical training is at the bedside. Traditionally, much of medical education has consisted of the use of passive learning techniques, such as lectures and rounds scattered among patient encounters. However, experiential learning theory [30], which is corroborated by the success of problem-based learning [31], holds that learning should be more directly linked to clinical encounters. This grounding automatically creates a curriculum that is germane to clinical practice and thus increases learner motivation. Research has shown that physicians in training [32] and in practice [12] generate abundant questions while caring for patients. Unfortunately, many of these “teachable moments” are missed because attending physicians do not seek answers or refer to convenient but less than reliable information resources [12, 33, 34]. Expeditious ways of seeking medical evidence, such as using systematic reviews, may increase the amount of learning done during clinical work and may improve clinical care at the same time.

Teachers can accomplish much by modeling the behavior we want our trainees to adopt, at the bedside or in the classroom (Table 1). We should model thoughtful searching for evidence rather than the “flash” retrieval of medical trivia from memory. A thoughtful approach teaches trainees both the value and the pleasure of searching for answers rather than maintaining a static knowledge base. Expediting the search for evidence by referring to systematic reviews strikes a balance between scientific rigor and the time pressures of clinical practice. We suggest the following examples of effective modeling.

At prepared conferences, teachers should cite systematic reviews whenever possible as sources of information. If none exist, a teacher may still stress their importance by noting this lack. In less structured situations, such as on ward rounds or in clinic, faculty should also model the use of systematic reviews. When questions arise that require less familiar information, the teacher can set about pursuing the evidence as described in the clinical scenarios at the beginning of this paper. If a current systematic review is not readily available in a textbook or article file, the teacher can try to locate one by using a previously saved literature search strategy, such as the computer-based strategy described by Hunt and McKibbon [35]. If the answer is not needed immediately, someone (including the teacher) can be assigned to bring a systematic review to the next session. We have found it efficient to place in our teaching computers a previously saved search strategy for locating systematic reviews. Trainees can download the strategy onto a floppy disk for use at home. Librarians can, of course, provide important help in searching for and in teaching persons how to search for systematic reviews. The librarians at the University of Rochester and at our institution have agreed to place previously saved search strategies, including the strategy for locating systematic reviews, on a network server so that they are readily available to users on- and off-campus.

The following methods may facilitate the use of systematic reviews in your practice and teaching (another article in this series discusses some of these methods in more detail [35]). First, work to increase the availability of systematic reviews in all clinical settings. We suggest that teachers obtain and use the resources cited in the (Table 1). For example, many clinical sites already have files of frequently used articles. Systematic reviews should be included to the maximum possible extent in such files. Second, compilations of systematic reviews published as books or monographs, such as those by the U.S. Preventive Services Task Force [4] (Table 1), are convenient if purchased but difficult to locate in journals. Another way to access systematic reviews is through the use of nonprint media. For example, the Internet provides access to the U.S. Preventive Services Task Force, the Agency for Health Care Policy and Research, and Canadian Guidelines (Table 1). Accessing the Internet quickly during clinical care is difficult to do without experience and an excellent dedicated computer system, but some reviews are available on disk. The Cochrane Library and the American College of Physicians position papers, which are accompanied by systematic reviews, can be purchased through the College. The College also sells Best Evidence, a CD-ROM that contains electronic versions of ACP Journal Club and Evidence-Based Medicine. Physicians can search this CD-ROM for meta-analyses, a common type of systematic review (Table 1).

As noted above, physicians can incorporate systematic reviews into training by encouraging residents to assist in performing one to fulfill the requirement for scholarly activity during training [36]. This approach would enable trainees not only to gain clinical expertise in a given topic but to learn the methodology of systematic reviews. This will lead them to better understand the strengths and weaknesses of systematic reviews and how to apply these reviews in clinical problem solving.

Barriers and Possible Solutions to the Use of Systematic Reviews in Medical Education

Lack of faculty support is a major potential barrier to the use of systematic reviews in training. Nongeneralists tend to value these reviews less than generalists do [12]. Furthermore, some teachers may feel threatened by systematic reviews, some of which may challenge their own opinions [37]. In addition, faculty may not be convinced of the value of systematic reviews. The following approaches may reduce faculty (and trainee) reluctance to use systematic reviews.

First, it is critical to emphasize that systematic reviews are not meant to replace clinician decision making. Rather, clinicians should be encouraged to become authorities on the clinical interpretation and application of systematic reviews. Second, faculty must be convinced of the value of these reviews. Unfortunately, no studies have examined the impact of systematic reviews on medical education. One study [38] found that practicing physicians were not particularly interested in using systematic reviews. However, this study is now several years old, and the exponential increase in the publication rate of systematic reviews means that they are now available for many clinical questions. When interpreting the results of this study, one should also realize that educating practicing physicians is generally not an effective way to change clinical practice [39]. This may support the use of systematic reviews during training because trainees may be more open to learning new ways of learning. Faculty may be less reluctant to use systematic reviews if they realize that producers of evidence-based clinical guidelines [4, 40], “opinion leader” physicians (leaders in hospital staffs, medical communities, and professional societies and members of certification and editorial boards) [12], policymakers for health care plans [41], and even the insurance industry [42] seek evidence from systematic reviews. We have observed that clinicians best learn the value of systematic reviews when they can readily locate one during a “teachable moment.”

Faculty use of systematic reviews may be increased through use of the following strategies. First, subspecialty teachers can be recruited into locating pertinent systematic reviews for inclusion in teaching files (many of these reviews are published only in subspecialty journals). A jointly prepared bibliography that emphasizes systematic reviews can then be provided to trainees before subspecialty rotations are started. Second, trainees can serve as “vectors,” spreading interest in and knowledge about the power and utility of systematic reviews. We can encourage trainees to solicit systematic reviews from all faculty, not just those who volunteer. Third, if a training program encourages residents to do systematic reviews to fulfill a requirement for scholarly work, the residents should solicit help from a wide variety of clinical faculty whose experience makes them content experts.

A second barrier is that trainees themselves do not understand the nature of systematic reviews. At our institution, we have modified our journal clubs so that they teach the strengths and limitations of systematic reviews. In addition, we teach brief guidelines on assessing the quality of systematic reviews [43], on distinguishing systematic from traditional reviews, and on efficiently searching MEDLINE for systematic reviews.

Authors of systematic reviews have an important role to play in facilitating the use of their work by trainees. Some systematic reviews are as tedious to read as original research articles. Authors should make their reviews easier to read and should use structured abstracts. Authors of systematic reviews, editors of medical journals, and custodians of such bibliographic databases as MEDLINE must continue working to improve the identification of systematic reviews and their distinction from traditional reviews. For example, MEDLINE cannot recognize some excellent systematic reviews [40] because they do not have abstracts [44], Medical Subject Headings, or words in the titles that distinguish them from traditional reviews.

Several barriers may hinder trainees who are interested in performing systematic reviews. Possible solutions are discussed in detail elsewhere [45]. Most important, residents need such resources as protected time, mentors, and methodologic instruction. Each training program should provide administrative time for a faculty member who is a designated coordinator of resident scholarly activity. This person can help trainees select feasible research questions and locate appropriate faculty mentors. Performing a systematic review correctly can be a demanding, complex, and frustrating exercise, and residents should not be expected to work alone on such a project. Thus, the coordinator should ensure that trainees receive both content-related and methodologic guidance. If the coordinator has expertise in performing systematic reviews, he or she can assist mentors who do not have experience. Alternatively, a subset of mentors could be trained by local experts or by programs such as the Cochrane Training workshop [46]. Finally, the coordinator should encourage clinician educators to be mentors and should acknowledge the help of mentors to department chairpersons.

Conclusions

Systematic reviews are becoming-and should become-integral to the dissemination of medical knowledge to physicians in training and in practice. Systematic reviews organize the medical literature and, hence, provide an interface between the physician and original research. Teaching and modeling the use of systematic reviews will therefore improve physician training and true lifelong professional development. We believe that the acquisition of such skills will ultimately result in better patient care. (Table 2)

Appendix

To identify references used in this paper 1) MEDLINE textword searches were done for each of the following: journal club, critical appraisal, grand rounds, dissemination, problem based learning; and 2) MEDLINE Medical Subject Heading searches were done separately for each of the following: databases, bibliographic and databases, electronic. In addition, a strategy similar to that described in [35] was combined with “septic shock.tw.” or “diethylstilb$.tw.” The 12/95 CD-ROM version of ACP Journal Club was searched for “meta-analyses.” Additional sources included the following document: National Library of Medicine. Current bibliographies in medicine: meta-analysis. Washington, DC: US Gov Pr Office; 1993.