How Consumers and Policymakers Can Use Systematic Reviews for Decision Making

Current Use of Systematic Reviews by Consumers and Policymakers

Our search for evaluations of the use of systematic reviews identified little published evidence to support the opinion that systematic reviews currently influence the medical or health care decisions made by the general public and by policymakers (details of our search strategy are available by contacting Dr. Bero at the address listed at the end of text). We found only one study [3] in which systematic reviews influenced hypothetical decisions about reimbursement for mammography screening and cardiac rehabilitation and one case report [4] in which one of the authors conducted a systematic review that persuaded a physician to change his recommendations. The lack of research on the impact that systematic reviews have on decision making may be the result of a lack of interest by the research community or the complexity of studying decision-making processes.

In our search, we attempted to identify studies that assessed the direct impact of systematic reviews on decisions made by policymakers and consumers. Although we only found two evaluations, the literature does offer numerous examples of how systematic reviews have been used to gather information for policymaking. For example, Light and Pillemer [5] describe how systematic reviews have been commissioned by policymakers to answer their questions. Guidelines on clinical practice (such as those from the Agency for Health Care Policy and Research and from the American College of Physicians) are often based on systematic reviews. In addition, technology assessments (such as those conducted by the U.S. Office of Technology Assessment) often include a systematic review of the literature on clinical efficacy as part of the assessment. We have learned that systematic reviews are more frequently cited than original research articles in coverage by the news media of research on the effects of environmental tobacco smoke; this fact suggests that systematic reviews might be indirectly influencing policy decisions as a result of such coverage [6]. The use of systematic reviews in policy development reinforces the need to rigorously evaluate their direct impact on policy decisions.

Several factors may explain the reason that minimal data are available on the impact of systematic reviews on decisions made by policymakers and consumers. Decision makers consider the source, format, perceived relevance, and other aspects of information when making decisions (Table 1). The tendency of decision makers to use anecdotal aspects of the most recent evidence or personal experience rather than evaluate evidence broadly and systematically undermines the use of systematic reviews [7]. In addition, the role of information depends on its interaction with other components of the decision-making process (including the values, preferences, and beliefs of the decision maker) and the context in which the decision is being made (Table 1) [8]. Furthermore, although the methods for conducting systematic reviews have been available to the medical community for years, these reviews have only recently been applied to clinical care [12-14]. For example, a landmark article summarizing the state of the science of systematic reviewing was published in the medical literature in 1987 [15]. In addition, the Cochrane Collaboration, an international organization whose goal is to design, conduct, and disseminate systematic reviews in medicine, was founded in 1992 [16].

The role of information as only one aspect of the complex decision-making process is illustrated by our hypothetical examples. The healthy pregnant woman who was deciding whether to have ultrasonography should be interested to learn that four systematic reviews that assessed routine ultrasonography in early pregnancy have found the procedure to be safe and effective for detecting fetal malformations [17-20]. However, she will probably weigh this information against her perceived risk for having a baby with a malformation, the amount of time she must take from work, the inconvenience associated with having ultrasonography (for example, out-of-pocket expenses), and the experience of her sister or next-door neighbor [21]. In contrast, the members of a city council, while making their decision on whether to restrict tobacco smoking, can be informed by three meta-analyses of the effects of passive smoke on heart disease [22-24]. However, council members are also likely to consider the opinions of their constituents and the pressures exerted by lobbyists for the tobacco industry and advocacy groups.

The Rationale for Using Systematic Reviews

Although information in any format plays only a limited (but potentially significant) role in the decision-making process, we have reason to believe that systematic reviews can have a particularly important influence on the decisions made by both consumers and policymakers. A properly conducted systematic review can provide an objective summary of large amounts of data. For consumers and policymakers who are interested in the bottom line of evidence, systematic reviews can help cohere conflicting results of research. Systematic reviews can form the basis for other integrative articles produced by policymakers, such as risk assessments, practice guidelines, economic analyses, and decision analyses [25]. Systematic reviews can aid the process of consensus development by curtailing the criticism that consensus development tends to occur in the absence of an objective framework for collecting and reviewing evidence [1]. Systematic reviews also typically identify gaps in knowledge, thereby helping consumers and policymakers decide not to proceed in the absence of evidence or encouraging them to address the gaps in medical research. Savulescu and colleagues [26] have recommended that medical ethics committees require researchers to conduct systematic reviews of existing relevant research to ensure the need for a new study.

As is true for the results of any research, however, inappropriate use of systematic reviews can result in more harm than good. Some of the risks of systematic reviews can be illustrated with our hypothetical example of the healthy pregnant woman. In the United Kingdom, leaflets that are targeted to pregnant women and health care professionals offer informed choice by reviewing the value of routine ultrasonography. The leaflets summarize systematic reviews of the best available evidence on the efficacy and safety of routine ultrasonography in pregnant women. In a case study on the reactions of women and health care professionals to the leaflets [27], women reacted with shock at the contents of the leaflets but were glad to be presented with the advantages and disadvantages of routine scanning and often requested additional information. Midwives believed that the leaflets would help women seek better health care, whereas ultrasonographers were concerned that the leaflets would provoke anxiety among women and lessen the use of routine ultrasonography [27]. The conflicting reactions of the midwives and ultrasonographers could lead to confrontation and lack of trust.

Additional harm could result from misrepresentation of the conclusions of systematic reviews to promote the self-interests of organizations or to support political positions. For example, a systematic review [24] of the cardiac effects of environmental tobacco smoke was presented without referring to other literature on the effects of passive smoking and was misconstrued as failing to conclude that environmental tobacco smoke is harmful [28]. Misinterpretation of systematic reviews in the lay literature can affect the decisions made by government officials, including our hypothetical example of a city council regulating tobacco smoking.

Although the potential benefits of systematic reviews seem to outweigh the harms, sustained efforts are needed to increase our understanding of each stage involved in the use of systematic reviews as a decision-making tool (Table 2). In the following sections, we describe strategies for optimizing the use of systematic reviews by using a model analogous to that proposed for the study of the diffusion of innovations [9].

Optimizing the Use of Systematic Reviews

Increasing Access to Systematic Reviews: Awareness and Identification of Reviews

For systematic reviews to influence decisions, health care consumers and policymakers must first be aware of the existence, characteristics, and potential value of the reviews. Policymakers are usually knowledgeable about research that supports their policy positions [7]. With increasing access to the Internet, consumers are also being exposed to information that was previously inaccessible [29]. It is unknown, however, how much consumers and policymakers know about systematic reviews, including where to find them.

Systematic reviews are often published in medical journals that can be identified through searches of bibliographic databases. Accessibility to systematic reviews could be improved if the reviews were available in the offices of policymakers, through public libraries, or by home computers. In addition, physicians could play a role in facilitating shared decision making by linking consumers to information in systematic reviews. One example of a nationwide effort to increase availability of reviews is practiced in the United Kingdom, where all hospitals, ambulance services, and community health care centers subscribe to the Cochrane Library, an electronic database of reviews published by the Cochrane Collaboration [30]. Thus, health care policymakers in the United Kingdom have access to systematic reviews. The distribution of reasonably priced collections of systematic reviews by public and professional organizations could also increase awareness of reviews.

Networks of interested persons can improve access to systematic reviews. The Cochrane Consumer Network is an international network that facilitates dissemination of information from systematic reviews to consumers, their families, and their advocates by sharing information among network members and notifying the news media of interesting developments [31]. (For information on the Cochrane Consumer Network, contact Hilda Bastian, Coordinator, Consumer Network, The Australasian Cochrane Centre, Flinders Medical Centre, Bedford Park SA 5042, Australia. Telephone: 61 8 204 5399; fax: 61 8 276 3305 [e-mail: hilda.bastian@flinders.edu.au].) A network of clinical policymakers from key organizations has been proposed as a way to increase awareness of systematic reviews of pregnancy and childbirth among health care professionals [32].

Critical Evaluation of Systematic Reviews

All research may be perceived as being driven by the biases and vested interests of the researchers and funders [33]. Systematic reviews, like other forms of research, vary in methodologic quality; biased reviews could lead to potentially damaging conclusions [34, 35]. Therefore, health care consumers and policymakers should consider limiting their sources of information to the most rigorously conducted systematic reviews. Simple tools for critically appraising reviews and databases might include distinguishing systematic reviews from narrative reviews [36]. Complete disclosure of funding sources and realization that authors may have conflicts of interest could also help consumers and policymakers select balanced systematic reviews [37]. For example, in the hypothetical example of a city council making decisions about enacting restrictions on tobacco smoking, council members should know that some review articles on the health effects of passive smoking were sponsored by the tobacco industry. These articles found a lack of adverse health effects (even after controlling for quality or peer review) in contrast to the findings of review articles that were not sponsored by the tobacco industry [38].

Using Systematic Reviews for Decision Making

Even if health care consumers and policymakers are given understandable, timely, relevant, objective information, numerous other factors (such as those listed in Table 1) influence the incorporation of systematic reviews into the decision-making process. Barriers to the use of evidence when making decisions have been discussed extensively by other authors and are beyond the scope of this article [21, 39, 40]. In brief, the use of systematic reviews can be increased relative to the use of personal value and contextual factors in making policy decisions by identifying the barriers that are specific to each target audience and designing focused strategies to overcome the barriers [10].

Participation in the Design and Reporting of Systematic Reviews

One major barrier to the use of systematic reviews by policymakers and consumers is that the information must be perceived as relevant to the decision at hand [21]. Health care consumers and policymakers may regard academic research as irrelevant because it is not timely, asks the wrong questions, or is simply not interesting [41]. Furthermore, because the research community controls the funding of a study, the manner in which the study is conducted, and the publication and dissemination of findings, consumers and policymakers have little opportunity to provide input.

The questions of researchers tend to differ from those of consumers and policymakers. To develop a focused research question, researchers may oversimplify the problem [42]. In contrast, consumers and policymakers prefer to focus on more complex questions that address problems in the context of their local circumstances. In addition, researchers may concentrate on measurable, intermediate outcomes that are irrelevant to policymakers and consumers. For example, outcomes that are relevant to persons who have heart failure might include mortality and health-related quality of life, but such outcomes as echocardiographic measurements or exercise-tolerance testing may not be of concern [43]. Research often does not provide data on the relative costs of treatments or on direct comparisons of treatments. Some of the deficiencies of research could be addressed by involving health care consumers and policymakers in the design of primary studies [44, 45]. For example, patients with cancer, their family members, and clinicians have worked together to identify, evaluate, and draw conclusions from relevant research, including systematic reviews, for developing a guide to the treatment of pain associated with cancer [46].

Systematic reviews with narrow research questions can assist policymakers and consumers if sufficient information has been provided to allow the decision maker to interpret the results. Systematic reviews that provide detailed information on the inclusion and exclusion criteria for original studies and an assessment of the characteristics of these studies can help consumers and policymakers determine whether the findings are helpful. For instance, in our example of a healthy pregnant woman who needs to decide whether to have ultrasonography, the women would be interested only in considering the results of systematic reviews that evaluated studies on the effects of ultrasonography in healthy women who had low-risk pregnancies.

Participatory approaches to research could produce systematic reviews that answer questions of greater relevance to consumers and policymakers than the questions now being addressed. The focus of participatory research is knowledge for action in contrast to knowledge for understanding [47]. Participatory research uses a bottoms-up approach to planning that considers local priorities, processes, and perspectives. For example, the National Health Service in the United Kingdom has tried to incorporate policy and consumer input into research planning [48]. Since 1991, the Service has conducted widespread consultations that have encouraged its own staff, statutory agencies, clinical management, professional bodies, consumer groups, academic centers, and research organizations to identify the needs for research, including systematic reviews.

Consumer and policymaker participation cannot, of course, correct the inadequacies that exist in the original research studies included in a systematic review. However, input from consumers and policymakers could ensure that the systematic review contains relevant information. In addition to commissioning topics for reviews (as encouraged by the National Health Service in the United Kingdom), consumers and policymakers could be educated to evaluate reviews critically and could use this capability to experiment with methods of interpreting and presenting the findings of systematic reviews. The Cochrane Collaboration provides unmatched opportunities for health care consumers to become involved in the design, interpretation, and use of the results of systematic reviews. Members of the Cochrane Consumer Network sit on the steering group of the Collaboration and, in increasing number, are joining the groups that are responsible for producing the reviews. Members of the Network are writing summaries of reviews that are targeted to a lay audience and are developing methods of disseminating the reviews to consumers. This involvement of consumers in such organizations as the Cochrane Collaboration could lead not only to more relevant research but also, in principle, to more efficient use of health care resources [31, 49].

Other factors that can influence decision making include how risk is reported [3] and the overall presentation of research results. The length of a report, the complexity of the language, the measurements used to express the outcomes, and the ease with which the bottom line is identified also can influence decision making. Executive summaries of systematic reviews that are developed specifically for (and, ideally, with) consumers and policymakers and that clearly and concisely state the results of these reviews could be useful in conveying their specialized information. The manner in which the summary is written should be intellectually accessible, relevant, and appealing. An example is the leaflets (developed in the United Kingdom) that summarized the result of systematic reviews of routine ultrasonography for health care consumers and professionals [27]. Another potentially useful strategy would be electronic publication of jargon-free versions of systematic reviews with hypertext links that allow readers who have various academic backgrounds to access information at different levels of complexity.

Conclusion

Systematic reviews can be very useful decision-making tools by objectively summarizing large amounts of information, identifying gaps in medical research, and identifying beneficial or harmful interventions. To date, little published research has evaluated the direct impact of systematic reviews on the decisions made by health care consumers and policymakers. This scarcity of research could be explained, at least partially, by limited availability of reviews; lack of interest by researchers; lack of evidence that the information would be beneficial; and absence of strategies that increase the awareness of consumers and policymakers and improve the identification, appraisal, and application of systematic reviews. Ideally, when decision makers incorporate results of systematic reviews in their decisions, they would collaborate with researchers to assess the impact of this information on outcomes [9].

Making decisions on health care policies and issues is a complex process in which obtaining information plays only one part of the whole. Even if all existing barriers to the use of systematic reviews were overcome, these reviews would be an important, but not an obligatory or sufficient, component of decisions on health care [50, 51]. Further efforts are required to improve our understanding of how various sources of information interact and how information is processed in different contexts by individuals with different values. In the meantime, efforts should continue to ensure that systematic reviews contain sufficient information on the reviews' methods and sponsorship and that they are presented in an accessible and appealing fashion that enables users to judge their validity and relevance.

Many of the current barriers to the use of systematic reviews by consumers and policymakers also affect clinicians and researchers. Time and effort would be wasted if each of these groups independently developed and evaluated strategies to overcome the common barriers. Finding the answers to the questions that are relevant to policymakers and consumers requires effective collaboration among all decision makers. (Table 3)