Series Editors:

Cynthia Mulrow, MD, MSc

Deborah Cook, MD, MSc

Historically, implicit clinical policies rested primarily with individual practitioners, protected under the rubric of "the art of medicine" and modulated by knowledge, experience, and heuristics [1]. Dissemination of information through peer-reviewed publications and traditional continuing medical education represent two major, albeit somewhat passive and indirect, attempts to inform clinical practice. The promulgation of explicit clinical policy embodied in practice guidelines has recently heightened awareness of the determinants of medical decision making [2].

Practice guidelines have been developed to improve the process of health care and health outcomes, decrease practice variation, and optimize resource utilization [3, 4]. Described as "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances" [5], guidelines attempt to distill a large body of medical expertise into a convenient, readily usable format. Practice guidelines based on the synthesis of the best, most recent evidence can help practitioners keep current with the literature and help them assimilate evidence into practice [6, 7].

The term "guideline" is used loosely to describe documents with different purposes, such as regulation of hospital admissions, use of tests and technology, transfer of seriously ill patients [8], and training programs [9]. Sporting such names as "practice policies," "practice parameters," and "clinical indicators" [10, 11], other decision aids further expand this lexicon. Clinical practice guidelines represent specific decision nodes that can be linked together to form clinical pathways or algorithms. Clinical pathways organize, sequence, and time the care given to a "typical, uncomplicated patient" [4, 12], whereas clinical algorithms are a set of more complex instructions for addressing a particular issue in which decisions and their consequences are expressed in conditional, branching logic [13].

This article focuses on the relation between systematic reviews and practice guidelines: how the development of guidelines can benefit from systematic reviews, and how systematic reviews can be used to help implement guidelines.

Methods for Developing Guidelines: An Overview

Methods used to develop guidelines differ according to the stakeholders involved, the degree of reliance on formal literature reviews, the extent to which expert opinion prevails, and the process by which the ultimate recommendations are expressed [11]. A multidisciplinary team may result in a holistic approach and wider endorsement of the final product [7]. Group processes commonly used to generate clinical recommendations include informal peer committees, nominal group techniques [14], the Delphi method [15], and expert or nonexpert consensus conferences. These strategies are not mutually exclusive, and their advantages and disadvantages have been summarized elsewhere [16]. One method for guideline development is outlined in the (Figure 1). Whether guidelines are created by local health providers, regional or national professional bodies, payers, or purchasers, the resources available may also determine the focus and methods. Few studies have compared the process, products, and health outcomes of these different approaches to developing recommendations [17].

View larger version (32K): [in this window] [in a new window]   Figure 1. One approach to developing guidelines and pathways on the basis of summaries of the relevant research evidence [12]. After defining the scope of the guideline and current practice, multidisciplinary teams search for, select, critically appraise, and update published systematic reviews or create their own systematic reviews of the primary literature when none exist [7]. The research is then classified according to its rigor, and recommendations are graded accordingly. Systematic reviews are used to select the most effective methods of guideline and pathway implementation, adapted to local circumstances. Updated systematic reviews, lessons learned through the implementation experience, and analysis of process measures and clinical and economic outcomes are then used to reformulate guidelines and pathways as a method of continuous quality improvement.

Selecting the clinical problem to be addressed by a guideline involves considering the prevalence of the problem, the clinical and economic burden it imposes, the resources available for its care, the availability of evidence for both existing care and improved care, and the likelihood of influencing practice [18, 19]. The next steps involve learning about current clinical practice as a baseline for change, defining the goals of the guidelines, and searching for the relevant research evidence (Figure 1). Because a guideline based on an incomplete or biased evaluation of the literature can lead to inappropriate recommendations, the search for relevant research should be comprehensive, research should be selected by using explicit criteria, and the validity of the results should be judged in a rigorous and reproducible fashion. Guideline and pathway developers ideally search for, select, critique, and combine data in a manner analogous to that used for a systematic review.

In addition to incorporating evidence and acknowledging its absence, creating clinical recommendations requires making value judgments about preferred courses of action. Some guidelines lay out choices and are distinctively influenced by patient preferences. Consider the difference in this regard between offering men with prostate cancer the choice of conservative management or radical prostatectomy (which builds patient preference into the decision) and assuming that all patients without advance directives who sustain a cardiac arrest want cardiopulmonary resuscitation (which does not).

Guideline documents ideally indicate how disagreements were handled and how information was synthesized (for example, by a qualitative pooling of opinion, a quantitative approach such as meta-analysis or decision analysis, or some combination of methods [19]). If guideline developers do not indicate how they identified and summarized the evidence and integrated different values, clinicians cannot adequately evaluate the rigor of the guidelines and the extent to which research evidence supports the recommendations [20, 21]. Recent guidelines for the diagnosis and treatment of idiopathic thrombocytopenic purpura [22] described the literature, emphasized the paucity of evidence and the limitations of opinion-based recommendations, and issued recommendations on the basis of clear documentation of the strength and variance of opinion [23].

The Role of Research Evidence in Practice Guideline Development

Guideline developers who want to incorporate research evidence into their clinical recommendations recognize the challenges of exhaustively searching the literature. For many common conditions, the volume of clinically useful literature is considerable. For other conditions, data may be sparse. Because guideline development is often limited by the difficulty of locating and appraising primary research, searching for and conducting systematic reviews is important. Accordingly, both primary research and integrative articles are potentially useful to guideline developers (Figure 1).

Systematic reviews are the most common type of integrative article. Their authors have searched for, selected, and synthesized (either qualitatively or quantitatively) evidence on specific clinical questions. Economic analyses quantitatively compare the costs and consequences of alternate courses of action. Decision analyses present the probability of various outcomes in terms of the values of expected benefits and harms of key decisions. Although clinical recommendations may emerge from these documents, their content is typically structured in a research report format. In contrast, practice guidelines are more often presented in a framework congruent with decision making; they suggest or support specific clinical recommendations but may reflect the scope of information contained in systematic reviews, economic analyses, and decision analyses.

For example, the American College of Physicians guidelines for magnetic resonance imaging of the brain and spine [24] were produced after a systematic review of neuroimaging with magnetic resonance imaging [25], and the guideline developers acknowledged the absence of studies comparing magnetic resonance imaging with other technologies. Relevant reviews are sometimes published after rather than before guidelines are developed. Such was the case with the 1994 clinical practice guideline on unstable angina by the Agency for Health Care Policy and Research [26], whose expert panel recommended that patients with unstable angina who are receiving aspirin should be treated with heparin for 2 to 5 days unless heparin is contraindicated. Two years later, a supporting meta-analysis showed a 33% reduction in risk for myocardial infarction or death in patients who had unstable angina and received aspirin plus heparin compared with those who received aspirin alone [27].

The scope of guidelines may reflect the particular interests of the guideline developers. The College's guidelines for the medical treatment of stroke prevention [28] clearly incorporate the results of a concurrently published systematic review [29] but are not explicit about cost-effectiveness issues (such as the rationale for choosing aspirin rather than ticlopidine). Other guidelines, such as those created by the American Heart Association for carotid endarterectomy [30], comprehensively summarize research and address economic issues more directly.

Examples of How To Use Systematic Reviews for Building Practice Guidelines and Clinical Pathways

As an alternate or complementary approach to formally summarizing primary research, guideline and pathway developers can use previously published systematic reviews that summarize the relevant primary studies. For the management of bleeding esophageal varices, approximately 200 randomized trials of pharmacologic, mechanical, and surgical interventions could be considered, as could a more manageable number of systematic reviews of these topics. For example, systematic reviews of primary prevention of variceal gastrointestinal hemorrhage indicate that ß-blockade and sclerotherapy reduce bleeding and mortality rates [31-33]. Systematic reviews also indicate the following. First, for control of acute bleeding, vasopressin is superior to no treatment but somatostatin is more effective than vasopressin [34, 35]. Second, although emergency sclerotherapy controls bleeding and decreases rebleeding rates more than do drug treatment and balloon tamponade [35], ligation is more effective than sclerotherapy [36]. Third, for the prevention of rebleeding, both sclerotherapy and ß-blockade are beneficial, but they are more effective in combination than either one is alone [35]. Finally, although shunt surgery decreases the risk for bleeding when performed prophylactically and decreases the risk for rebleeding when done as treatment, surgery markedly increases the risk for hepatic encephalopathy and may hasten death in both settings [35]. In summary, persons developing clinical pathways who are interested in the broad spectrum of decisions associated with long-term management of variceal bleeding have a solid database of well-conducted systematic reviews of rigorous trials on which to build.

A focused initiative based on results of one systematic review might involve the creation of a single guideline. Investigators at Cedars-Sinai Medical Center in Los Angeles, for example, concentrated on early endoscopy and a short hospital stay for certain "low-risk" patients with nonvariceal gastrointestinal hemorrhage. A systematic review indicated that early therapeutic endoscopy has a favorable effect on rebleeding, surgery, and mortality [37], suggesting that timely risk stratification and treatment chosen according to clinical and endoscopic features would allow clinicians to consider early discharge of patients at low risk for adverse events. The promising results of a retrospective validation study that examined the effect of a guideline recommending this approach [38] were confirmed by a prospective time series study. This study showed that management of patients in accordance with the guideline resulted in no difference in morbidity or mortality but led to greater patient satisfaction, shorter hospital stays, and lower costs [39].

Limitations of Relying on Systematic Reviews in Guideline Development

Although systematic reviews of treatment can lay the foundation for practice guidelines, they are not a panacea. The most fundamental limitation of relying on reviews is that they never obviate the need for at least some critical appraisal of the original studies to understand the populations, interventions, and outcomes evaluated; the heterogeneity of these features; and the individual study results.

Considering our variceal bleeding example, trials of new treatment (for example, octreotide after variceal ligation [40] and combination ß-blocker and nitrate therapy [41]), as well as new surgical interventions, such as transjugular intrahepatic portosystemic shunting and liver transplantation [42], may not be summarized in systematic reviews. Even if guidelines are based on a careful systematic review of one approach (such as medical prevention of strokes [28]), the final recommendations may be narrowly focused (for example, by not considering carotid endarterectomy).

Not all outcomes of interest are measured in all primary studies; thus, certain measures, such as quality of life, are often inadequately represented in reviews. Because most randomized trials are underpowered for rare events or unusual adverse effects of treatment and may not report them, systematic reviews do not usually provide such information. Making diagnoses, triaging, and understanding patient preferences require alternate study designs; such material is less likely to be summarized in systematic reviews. The same holds for many other aspects of clinical practice based on pathophysiologic rationale and conventional wisdom. Obviously, sole reliance on systematic reviews will never adequately serve development of guidelines and pathways.

Systematic reviews and, therefore, the practice guidelines based on them may require modification in areas for which evidence continually emerges. For example, the results of a 1985 systematic review of randomized trials suggesting that histamine-2-receptor antagonists were beneficial in acute nonvariceal gastrointestinal bleeding [43] were questioned 7 years later by a large, more rigorous negative trial of a newer histamine-2-receptor antagonist, famotidine [44]. The postulate that acid suppression averts rebleeding was further challenged by another large negative trial of the proton pump inhibitor omeprazole [45]; this trial showed that the drug had no benefit in terms of rebleeding, need for surgery, or mortality. Emphasizing the vicissitudes of this situation is the most recent trial of high-dose oral omeprazole, which decreased continued or further bleeding and surgery in patients with ulcer [46].

Examining published reviews (and the primary research these reviews summarize) in preparation for guideline formulation often shows that research is insufficient to inform management decisions. Exposing gaps in medical knowledge can be a powerful stimulus for future research by guideline developers.

Finally, as the number of systematic reviews continues to increase, clinicians will be increasingly faced with more than one review on the same topic. Such multiple reviews may be helpful, particularly if later reviews update earlier ones, if one review resolves disagreements among previous reviews [47], of if concurrently published reviews yield concordant results. However, multiple systematic reviews may also challenge guideline and pathway developers if they generate conflicting conclusions. Strategies for dealing with such situations include careful critical appraisal of each review [48], with particular emphasis on issues of clinical and statistical heterogeneity [49], and a more detailed assessment of the relation between the focus of each review and the specific purpose of the guideline under development.

Using Systematic Reviews for the Format and Expression of Guidelines

To maximize the chance that guidelines reach the right practitioners at the right time and to help practitioners make the right decisions for the right patients, guidelines should be available at the place and time of decision making. Clinicians prefer short manuals or summaries of major recommendations and a synopsis of the underlying evidence that summarizes expected benefits and harms [50]. Publication of companion systematic reviews and executive summaries of their results may therefore be useful in guideline dissemination. Embedding guidelines in electronic medical records may also be helpful.

The strength of treatment recommendations is ideally informed by the quality of the research evidence; the magnitude, precision, and reproducibility of the treatment effect; and the relative value (determined by guideline developers, health care workers, and patients) of various outcomes. A key component of guidelines, therefore, is how accurately they reflect the inference conferred by the underlying research evidence. When all else is equal, recommendations about therapy are strongest when they are derived from systematic reviews of randomized trials that have consistent results, as opposed to reviews of trials or observational studies that have inconsistent results [51].

For example, the American College of Chest Physicians Antithrombotic Consensus Conferences classify evidence as level I (evidence obtained from randomized trials or meta-analyses of randomized trials, for which the lower limit of the CI around the point estimate of the treatment effect exceeds the minimal clinically important benefit), level II (evidence obtained from randomized trials or meta-analyses, for which the lower level of the CI around the treatment effect overlaps the minimal clinically important benefit), level III (evidence obtained from nonrandomized, concurrent cohort studies), level IV (evidence obtained from nonrandomized historical cohort studies), and level V (evidence obtained from case series or expert opinion) [52]. Thus, for example, the American College of Chest Physicians assigned a grade C recommendation to the use of long-term anticoagulation in patients with bioprosthetic valves and a history of systemic embolism; this grade is based on the use of level V evidence (expert opinion) in the formulation of the recommendations [53]. In this way, guidelines can avoid ambiguous terminology because readers are made aware of the basis for the recommendations and the extent to which the recommendations reflect the strength of the research evidence (Figure 1)

Using Systematic Reviews of Implementation Strategies for Practice Guidelines

Because the construction of guidelines alone does not change clinical practice, the next step is to disseminate and implement them (Figure 1). Some modern guideline documents describe how the guidelines were developed, but they rarely contain information on dissemination and implementation strategies. As described below, several systematic reviews [54-57] have summarized the studies that sought to determine which methods are most effective for changing clinician behavior and patient health status. Modification of provider practices and other "process measures" have been the outcomes most often studied. Although these may be viewed as less important surrogate outcomes, if selected carefully they can be viewed as intermediate steps that causally link recommendations to more meaningful patient-centered outcomes.

A systematic review of 59 rigorous studies evaluating the effect of practice guidelines found that 55 of them detected significant improvements in the process of care after introduction of the guidelines [54]. An update of this review [55] indicated that 12 of 17 studies assessing patient outcome after guideline implementation reported significant improvement in at least one outcome. Several insights emerged from this work. For example, the most effective strategies for implementing practice guidelines are educational interventions that are developed within local organizations and implemented through patient-specific reminders, whereas the least effective are externally developed guidelines that are disseminated through publications and implemented by general reminders.

These findings complement data summarized in another review of educational interventions targeted at changing the performance of practicing physicians and the health of their patients. Davis and colleagues [56] examined 99 randomized trials of continuing medical education. Almost two thirds of the trials showed an improvement in at least one outcome; 70% showed a change in physician behavior, and 48% of interventions aimed at health outcomes produced a positive change. Effective strategies included specific reminders, patient-mediated interventions, outreach visits (including academic detailing and opinion leaders), and other multifaceted activities. Formal continuing medical education sessions without practice-reinforcing strategies, educational materials, and audit and feedback were not found to be very effective. Another systematic review of the controlled studies evaluating computer-based clinical decision support systems showed that clinician behavior and patient outcomes can be modified by the use of these methods [57].

The less compelling the research evidence incorporated into guidelines, the greater is the need to test the impact of the guidelines on patient outcomes. Therefore, rigorous evaluation depends on understanding not just the guideline development process but also the specific methods used to disseminate and implement the guidelines. To date, systematic reviews have summarized single and multiple implementation methods. Although complex, multimethod approaches are frequently used, current knowledge of their effectiveness and generalizability remains somewhat limited. In making such evaluations, it is important to consider the health care setting in which the guidelines are implemented; valid practice guidelines coupled with effective implementation strategies may have no impact if constraints of access, availability, or cost are very strong or if attitudinal barriers prevent their endorsement [58]. Moreover, no matter how they are developed and implemented, guidelines may do more harm than good if they are inappropriately interpreted or applied.

Summary

Top Summary Author & Article Info References

In the past decade, the practice guidelines movement has become a major academic and commercial enterprise. If created by using the most valid and current research evidence summarized in systematic reviews, guidelines are one of many tools that can help to translate research evidence into clinical decision aids, optimize health outcomes, and educate clinicians. Like all decision aids, however, guidelines should be integrated with pathophysiologic reasoning and experience and should be adopted, adapted, or rejected according to patient preferences and the constraints of each health care setting. Practice guidelines and clinical pathways have potential limitations, many of which can be overcome by using an evidence-based approach in their development and by drawing on state-of-the-art implementation strategies that themselves have been summarized in systematic reviews. This field is ripe for future health services research.

Key Points To Remember

Formal reviews of the literature are fundamental to the development of sound practice guidelines

Developers of evidence-based guidelines generate their own systematic reviews and critically appraise and update previously published systematic reviews

When current systematic reviews are presented as companion documents to practice guidelines, they can help to communicate the evidence that supports specific clinical recommendations

Like all research evidence, data summarized in systematic reviews are important but not sufficient for practice guideline development; accordingly, recommendations should be interpreted in light of patient preferences and the health care setting in which the recommendations are implemented, which governs issues of feasibility and finance

Systematic reviews of the effectiveness of strategies for disseminating and implementing guidelines can be used to help select the approach that will produce the maximum impact on caregiver behavior and patient outcome

Drs. Greengold and Weingarten: Departments of Medicine and Applied Health Services Research, Cedars-Sinai Medical Center, Los Angeles, CA 90048.

Dr. Ellrodt: Department of Medicine, Division of General Internal Medicine, Cedars-Sinai Medical Center, Los Angeles, CA 90048.