RSS Feeds
Systematic Review: Surveillance Systems for Early Detection of Bioterrorism-Related Diseases
- Dena M. Bravata, MD, MS;
- Kathryn M. McDonald, MM;
- Wendy M. Smith, BA;
- Chara Rydzak, BA;
- Herbert Szeto, MD, MS, MPH;
- David L. Buckeridge, MD, MSc;
- Corinna Haberland, MD; and
- Douglas K. Owens, MD, MS
- From the University of California, San Francisco–Stanford Evidence-based Practice Center and Center for Primary Care and Outcomes Research, Stanford University School of Medicine, Stanford, California; Veterans Affairs Palo Alto Healthcare System, Palo Alto, California; and Kaiser Permanente, Redwood City, California.
Abstract
Background: Given the threat of bioterrorism and the increasing availability of electronic data for surveillance, surveillance systems for the early detection of illnesses and syndromes potentially related to bioterrorism have proliferated.
Purpose: To critically evaluate the potential utility of existing surveillance systems for illnesses and syndromes related to bioterrorism.
Data Sources: Databases of peer-reviewed articles (for example, MEDLINE for articles published from January 1985 to April 2002) and Web sites of relevant government and nongovernment agencies.
Study Selection: Reports that described or evaluated systems for collecting, analyzing, or presenting surveillance data for bioterrorism-related illnesses or syndromes.
Data Extraction: From each included article, the authors abstracted information about the type of surveillance data collected; method of collection, analysis, and presentation of surveillance data; and outcomes of evaluations of the system.
Data Synthesis: 17 510 article citations and 8088 government and nongovernmental Web sites were reviewed. From these, the authors included 115 systems that collect various surveillance reports, including 9 syndromic surveillance systems, 20 systems collecting bioterrorism detector data, 13 systems collecting influenza-related data, and 23 systems collecting laboratory and antimicrobial resistance data. Only the systems collecting syndromic surveillance data and detection system data were designed, at least in part, for bioterrorism preparedness applications. Syndromic surveillance systems have been deployed for both event-based and continuous bioterrorism surveillance. Few surveillance systems have been comprehensively evaluated. Only 3 systems have had both sensitivity and specificity evaluated.
Limitations: Data from some existing surveillance systems (particularly those developed by the military) may not be publicly available.
Conclusions: Few surveillance systems have been specifically designed for collecting and analyzing data for the early detection of a bioterrorist event. Because current evaluations of surveillance systems for detecting bioterrorism and emerging infections are insufficient to characterize the timeliness or sensitivity and specificity, clinical and public health decision making based on these systems may be compromised.
Key Summary Points
The practice of surveillance is changing to address the threat of bioterrorism and to take advantage of the increasing availability of electronic data.
The authors identified published descriptions of 29 systems designed specifically for bioterrorism surveillance.
Bioterrorism surveillance systems either monitor the incidence of bioterrorism-related syndromes (9) or monitor environmental samples for bioterrorism agents (20).
Only 2 syndromic surveillance systems and no environmental monitoring system were evaluated in peer-reviewed studies.
Both evaluations of syndromic surveillance systems compared the incidence of flu-like illness syndromes with results from national influenza surveillance.
Existing evaluations of surveillance systems for detecting bioterrorism are insufficient to characterize the performance of these systems.
Evaluation of bioterrorism surveillance is needed to inform decisions about deploying systems and to facilitate decision making on the basis of system results.
The anthrax attacks of 2001 and the recent outbreaks of severe acute respiratory syndrome (SARS) and influenza strikingly demonstrate the continuing threat from illnesses resulting from bioterrorism and related infectious diseases. In particular, these outbreaks have highlighted that an essential component of preparations for illnesses and syndromes potentially related to bioterrorism includes the deployment of surveillance systems that can rapidly detect and monitor the course of an outbreak and thus minimize associated morbidity and mortality (1-3). Driven by the threat of additional outbreaks resulting from bioterrorism and the increasing availability of data available for surveillance, surveillance systems have proliferated. The Centers for Disease Control and Prevention (CDC) defines surveillance systems as those that “collect and analyze morbidity, mortality, and other relevant data and facilitate the timely dissemination of results to appropriate decision makers” (3, 4). However, there is little consensus as to which sources of surveillance data or which collection, analysis, and reporting technologies are probably the most timely, sensitive, and specific for detecting and managing bioterrorism-related illness and related emerging infectious diseases (5).
Existing surveillance systems for bioterrorism-related diseases vary widely with respect to the methods used to collect the surveillance data, surveillance characteristics of the data collected, and analytic methods used to determine when a potential outbreak has occurred. Traditionally, the primary method for collecting surveillance data was manual reporting of suspicious and notifiable clinical and laboratory data from clinicians, hospitals, and laboratories to public health officials (6). Recent innovations in disease surveillance that may improve the timeliness, sensitivity, and specificity of bioterrorism-related outbreak detection include surveillance for syndromes rather than specific diseases and the automated extraction and analysis of routinely collected clinical, administrative, pharmacy, and laboratory data. Little is known about the accuracy of surveillance systems for bioterrorism and related emerging infectious diseases, perhaps because of the diversity of potential data sources for bioterrorism surveillance data; methods for their analysis; and the uncertainty about the costs, benefits, and detection characteristics of each.
Under the auspices of the University of California, San Francisco–Stanford Evidence-based Practice Center, we prepared a comprehensive systematic review that evaluated the ability of available information technologies to inform clinicians and public health officials who are preparing for and responding to bioterrorism and related emerging infectious diseases (7). In this paper, we present the available data on existing systems for surveillance of illnesses and syndromes potentially related to bioterrorism and the published evaluation data on these systems.
Methods
We sought to identify published reports of surveillance systems designed to collect, analyze, and report surveillance data for bioterrorism-related diseases or syndromes or reports of surveillance systems for naturally occurring diseases, if potentially useful for bioterrorism surveillance. We used the U.S. Department of Health and Human Services' definition of bioterrorism-related diseases (8-10). Because most patients with bioterrorism-related diseases initially present with influenza-like illness, acute respiratory distress, gastrointestinal symptoms, febrile hemorrhagic syndromes, and febrile illnesses with either dermatologic or neurologic findings, we considered these conditions to be the bioterrorism-related syndromes. We briefly summarize our methods, which are described in detail elsewhere (7).
Literature Sources and Search Strategies
We searched 3 sources for relevant reports: 5 databases of peer-reviewed articles (for example, MEDLINE, GrayLIT, and National Technical Information Service), government reports, and Web sites of relevant government and commercial entities. We consulted public health, bioterrorism preparedness, and national security experts to identify the 16 government agencies most likely to fund, develop, or use bioterrorism systems (for example, CDC and U.S. Department of Defense). We searched the Web sites of these government agencies and other academic and commercial sites. Finally, we identified additional articles from the bibliographies of included articles and from conference proceedings.
We developed 2 separate search strategies: 1 for MEDLINE (January 1985 to April 2002) and 1 for other sources. In both searches, we included terms such as bioterrorism, biological warfare, information technology, surveillance, public health, and epidemiology. Complete search strategies are available from the authors (7).
Study Selection and Data Abstraction
We reviewed titles, abstracts, and full-length articles to identify potentially relevant articles. Two abstractors, who were blinded to the study authors, abstracted data from all included peer-reviewed articles onto pretested abstraction forms. Given the large volume of Web sites screened, only 1 abstractor, whose work was frequently reviewed by a colleague, collected data from each Web-based report.
Evaluation of Reports of Surveillance Systems
The CDC developed a draft guideline for evaluating public health surveillance systems (3, 11, 12). This guideline recommends that reports of surveillance systems include the following: descriptions of the public health importance of the health event under surveillance; the system under evaluation; the direct costs needed to operate the system; the usefulness of the system; and evaluations of the system's simplicity, flexibility (that is, “the system's ability to change as surveillance needs change”), acceptability (“as reflected by the willingness of participants and stakeholders to contribute to the data collection, analysis and use”), sensitivity to detect outbreaks, positive predictive value of system alarms for true outbreaks, representativeness of the population covered by the system, and timeliness of detection (11, 12). The guideline describes these key elements to consider in an evaluation of a surveillance system but does not provide specific scoring or an evaluation tool. We abstracted information about each CDC criterion from each included reference.
Data Synthesis
We reviewed 17 510 citations of peer-reviewed articles and 8088 Web sites, of which 192 reports on 115 surveillance systems met our inclusion criteria (Figure 1). Of these, 29 systems were designed specifically for detecting bioterrorism-related diseases (as defined by the U.S. Department of Health and Human Services [8-10]) or bioterrorism-related syndromes (for example, flu-like syndrome and fever with rash). An additional 86 systems were designed for surveillance of naturally occurring illnesses, but elements of their design, deployment, or evaluations may be relevant for implementing or evaluating bioterrorism surveillance systems. For example, we included reports of systems for surveillance of nonbiothreat pathogens if they were designed to rapidly transmit surveillance data from sources that could be useful for detecting bioterrorism-related illness (for example, laboratory data, clinicians' reports, hospital-based data, or veterinary data) or if they reported methods of spatial or temporal analyses that facilitated rapid and accurate decision making by public health users. We present the evidence about the systems designed principally for bioterrorism surveillance systems and summarize the evidence about the other surveillance systems.
Surveillance Systems Designed for Bioterrorism-Related Diseases or Syndromes
We identified 2 types of systems for surveillance of bioterrorism-related diseases or syndromes: those that monitor the incidence of bioterrorism-related syndromes and those that collect and transmit bioterrorism detection data from environmental or clinical samples to decision makers.
Surveillance Systems Collecting Syndromic Reports
The 9 surveillance systems designed to monitor the incidence of bioterrorism-related syndromes vary widely with respect to syndromes under surveillance, data collected, flexibility of the data collection tool (for example, some Web-based systems allow remote users to change the prompts given to data collectors), acceptability to data collectors, and methods used to analyze the data (13-23) (Table).
Two syndromic surveillance systems were evaluated in peer-reviewed reports: the National Health Service Direct system and the program of systematic surveillance of International Classification of Diseases, Ninth Revision (ICD-9), codes from the electronic medical records of the Harvard Vanguard Medical Associates (20, 23). In these evaluations, the numbers of flu-like illnesses or lower respiratory tract syndromes detected by the syndromic surveillance system were similar to the national influenza surveillance data against which they were compared (20, 23). These published evaluation studies lacked information in several key areas: No reports characterized the detection capabilities of syndromic surveillance systems for nonpulmonary syndromes or provided specific information on any of these systems' acceptability, representativeness, or cost. Furthermore, we found no standard definitions for the syndromes under surveillance, and none of the included syndromic surveillance systems that rely on clinicians' entry of patient data defined the syndromes on the collection tool (for example, “flu-like illness” was not defined on the data entry screen or paper tool).
Some other promising systems that were not evaluated in peer-reviewed reports are currently being evaluated (22). These include the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE), which automatically downloads ICD-9 code data from U.S. Department of Defense health care facilities around the world and performs thousands of analyses daily (17). Other local systems, such as the tally sheet system used by the Santa Clara County Public Health Department, collect triage data from emergency department nurses and rely on manual data collection, analysis, and reporting; this information enables syndromic surveillance to occur in settings where electronic medical records are unavailable (22). The Rapid Syndrome Validation Project (RSVP) similarly relies on medically trained staff for collecting surveillance data (21). Physicians enter data on patients presenting with a syndrome of interest into a computer that has a touch-screen interface with RSVP. These systems are being evaluated for various surveillance characteristics, including determination of their sensitivity, specificity, timeliness, and acceptability.
Although most systems for syndromic surveillance are continuously collecting, analyzing, and reporting data, some systems are designed for short-term use at events thought to be potential bioterrorist targets (“event-based” or “drop-in” surveillance). For example, the Lightweight Epidemiology Advanced Detection and Emergency Response System (LEADERS) was used for syndromic surveillance at the 1999 World Trade Organization Summit and the 2001 Presidential Inauguration (19). This system requires staff at participating hospitals to complete a brief Web-based form after each initial patient visit describing the patient's syndrome and whether the patient participated in the event of interest. These syndromic incidence data can be monitored remotely by decision makers. Interpreting surveillance data from event-based surveillance systems can be complicated by the lack of adequate baseline data. For example, if an event-based surveillance system begins collecting surveillance data on 1 or more syndromes of interest a few weeks before the event, pre-event data may be insufficient to calculate an expected rate of cases for the weeks during and immediately after the event of interest. No evaluations of event-based surveillance systems have been published.
Surveillance Systems Collecting Environmental Detection Data
Appendix Table 1 presents the 20 detection systems that transmit data collected from environmental or clinical samples for analysis and presentation to remotely located decision makers. These systems differ in the type and location of sample collected (for example, aerosol samples continuously taken from locations in fixed sites, such as airports or public buildings; environmental samples taken from a site thought to be contaminated by a suspicious powder or other potential bioterrorism exposure; or clinical samples taken from potentially contaminated food, animals, or humans). These systems also differ in the specific technologies used to analyze the samples and send results to data warehouses for analysis and reporting. For example, The Interim Biological Agent Detector is used on U.S. naval ships to continuously monitor the air for a significant increase in particulate concentrations (32, 39-42). If a peak increase is detected, the instrument automatically collects an aerosol sample and alerts the ship's damage control center so the crew can collect and screen the sample with a handheld antigen test. Similar to this naval system, many detection systems were designed by the military and are now being adapted for civilian use. No peer-reviewed evaluations have described these systems; most were described only in government reports and Web-based information provided by manufacturers. None of these reports specifically described timeliness, necessary training, or security measures for specimens or surveillance data.
Surveillance Systems Designed for Other Purposes
Appendix Table 2 presents the 86 surveillance systems that were not designed for bioterrorism but are potentially relevant for bioterrorism surveillance. Each system is described in detail elsewhere (7). In this paper, we present general information about the types of systems, the evaluation data available about them, and their potential utility for bioterrorism surveillance.
Surveillance Systems Collecting Clinical Reports
The 6 surveillance systems that collect clinical information from networks of sentinel clinicians differ with respect to the diseases under surveillance, the frequency and method of reporting, the types of clinicians collecting data, and the timeliness of feedback to clinicians and health departments (55-79). Two of these systems—the French Communicable Disease Network and Eurosentinel—have been described in peer-reviewed evaluation reports (64, 65, 80). A retrospective evaluation of the French Communicable Disease Network found that the combination of clinicians' reports with information on viral isolates from the French Reference Centers was more timely than surveillance performed with viral isolates alone (65). The Eurosentinel project uses an international group of volunteer physicians who submit weekly reports to a coordinating center in Belgium. Outputs for influenza are available within minutes of reporting; however, data for other diseases are released in a quarterly newsletter (57). A report describing the first 3 years of the project found that discrepancies in disease-reporting practices, particularly the use of different denominators among the sentinel networks from different countries, made it difficult to compare the data from participating networks (57).
Surveillance Systems Collecting Influenza-related Data
Our search identified 13 systems for influenza surveillance (15, 81-101), of which 5 systems have been described in peer-reviewed evaluation reports (84, 87, 88, 90, 97). In general, these evaluations indicate that electronic reporting methods are more timely than manual systems (87, 88).
There is no clear consensus as to the most sensitive, specific, or timely data for influenza surveillance. An analysis of data from the Regional Influenza Surveillance Group of France found that sick-leave prescriptions, emergency house calls, and numbers of patients with influenza-like illness seen by general practitioners and pediatricians were the most sensitive indicators for the early recognition of influenza (87, 88). In contrast, data from the Viral Watch Program of South Africa suggest that viral isolates are more sensitive indicators of influenza activity than school absenteeism or mortality rates (84). A comparison of school absenteeism data collected by the Japanese School Health Surveillance System with data from the national influenza surveillance system demonstrated a sensitivity of 80% and a specificity of 100% (90). However, the authors noted that gaps in surveillance data during school holidays and the possible inclusion of non–influenza virus infections (for example, adenovirus) complicated the use of these data for influenza surveillance (90). These results do not provide sufficient evidence to favor the use of any given source of influenza data or method of collection or analysis.
Surveillance Systems Collecting Laboratory Data
Evaluations of systems for the surveillance of laboratory and antimicrobial resistance suggest that automated laboratory reporting systems are generally more timely and sensitive than conventional reporting methods (108, 117, 119, 120, 133). The sensitivity of these systems (typically compared with manual systems) ranged from 76% to 100% (117, 120); the specificity (95%) was reported for only 1 system (117). Few reports described methods for manipulating samples or confirming results, acceptability, or cost. No system was evaluated specifically for detecting a biothreat agent.
Surveillance Systems Collecting Foodborne Illness Data
We found 7 systems that collect and analyze reports from clinicians or laboratories about the incidence and characteristics of foodborne pathogens (139-150) and 3 systems that model microbial growth responses to food production methods (151-154). Evaluation data on these systems are limited to estimates of disease incidence identified by the systems but do not further describe the systems' sensitivity, specificity, or timeliness.
Surveillance Systems Collecting Zoonotic and Animal Disease Data
We found 2 systems for surveillance of zoonotic illnesses and 4 systems for the surveillance of animal diseases (155-166). None has been described in a peer-reviewed evaluation. Most reports provide little or no information about the timeliness of these systems; those that did suggest lag times are too long for effective bioterrorism surveillance.
Surveillance Systems Collecting Other Kinds of Data
We found 16 systems designed specifically for hospital surveillance (167-190). Evaluations of some hospital surveillance systems reported improvements in the timeliness and sensitivity of detecting nosocomial infections when compared with manual methods (168-170, 175, 176, 178, 183). An additional 12 systems met our inclusion criteria but did not belong in the preceding categorizations, including 6 systems that collect data about specific groups of patients (81, 86, 105, 191-196), 2 systems that collect pharmacy data (197, 198), and other systems (199-203). Evaluations of these systems generally showed little evidence that these systems have sufficient sensitivity, specificity, or timeliness to detect a bioterrorist event.
Evaluation of Reports of Surveillance Systems
When applying the CDC's guidelines for evaluating reports of surveillance systems, we abstracted whether the authors specifically described each characteristic of interest (Figure 2). The discussion of these characteristics was often modest and was based on opinion rather than formal evaluation (for example, some authors reported that the system under evaluation “was sensitive” without reporting actual sensitivity or specificity). Only 1 report addressed all 9 CDC criteria (90). Seventy-two reports of 43 systems described their timeliness, 29 reports of 22 systems described their sensitivity, and 15 reports of 12 systems described their specificity; however, only 12 reports of 9 systems described all 3 characteristics. Only 3 reports of 3 systems provided numeric data for both sensitivity and specificity of the system (90, 117, 175).
Discussion
Our systematic review identified 115 existing surveillance systems, 29 of which were designed for surveillance of illnesses and syndromes associated with bioterrorism-relevant pathogens. The evidence used to judge the usefulness of the reviewed systems is limited. Of the studies that evaluated systems for their intended purpose, few adhered to the CDC's published criteria for high-quality evaluations of surveillance systems. Even if a system was found useful for its intended purpose (for example, surveillance for influenza), we can only infer that the system might be useful for responding to bioterrorism.
Systems for bioterrorism surveillance require 3 key features: timeliness, high sensitivity and specificity, and routine analysis and presentation of the data that facilitate public health decision making. We discuss each characteristic in the following sections.
Timeliness
Effective surveillance for bioterrorism-related illness depends on systems that promptly collect, analyze, and report data to decision makers, because the effectiveness of intervention after a bioterrorism attack has been strongly linked to the rapidity of detection (204, 205). The evaluations of surveillance systems demonstrated 2 key factors affecting their timeliness. First, in general, the electronic collection and reporting of surveillance data improved detection compared with older, manual methods. Despite the advantages of electronic collection and reporting and the increasing availability of administrative and medical record data that can be transmitted instantaneously, many local health departments do not currently have adequate resources to manage, analyze, and interpret such large data sets. Also, as the size and complexity of the data under surveillance increase, so does the time required to analyze and interpret the data. Some systems that facilitate manual reporting of suspicious cases by clinicians and triage staff through fax or computer entry to public health officials may substantially reduce delays in reporting and represent programs that could be used in places without electronic medical records or electronic disease reporting or in health departments without extensive electronic data management resources. Surveillances systems must be evaluated to specifically delineate the time required for each step in the surveillance process from initial data collection to arrival of data at the health department to decision making about outbreak investigation.
Second, the timeliness of a surveillance system is affected by the source of surveillance data. For example, school and work absenteeism, calls to telephone care nurses, and over-the-counter pharmacy sales may provide earlier indications of bioterrorism than hospital discharge data or coroners' reports. Relatively few of the 115 included systems collect the earliest types of surveillance data—a potentially important gap in available surveillance systems. Systems that collect pharmaceutical data, such as EPIFAR (198), are promising for bioterrorism surveillance. Pharmaceutical data, particularly over-the-counter medication sales data, can indicate an outbreak, although these data would probably not be specific for bioterrorism. In addition, most pharmaceutical sales are tracked electronically. The detection characteristics of common prescription and nonprescription medications used for bioterrorism-related syndromes must be carefully analyzed to determine the utility of these data for bioterrorism surveillance. Similarly, surveillance systems must be evaluated to compare the timeliness of detection on the basis of the source of data used. Evaluations that determine how integration of several data sources affects the timeliness and accuracy of the system are also needed.
Sensitivity and Specificity
Bioterrorism surveillance systems with inadequate sensitivity may fail to detect cases of bioterrorism-related illness, which could result in substantial delays in detection and potentially catastrophic increases in morbidity and mortality. Systems with inadequate specificity may have frequent false alarms, which may result in costly actions by clinicians and public health officials or, perhaps even worse, officials ignoring the system when it reports a suspicious event. Because sensitivity and specificity are related, they must be evaluated simultaneously. However, only 3 reports of 3 systems provided numeric data for both sensitivity and specificity of the system (90, 117, 175). This substantially limits our understanding of the accuracy of existing surveillance systems for bioterrorism-related illness.
In addition, because there have been so few cases of bioterrorism-related illness, there are no reference standards against which to compare the surveillance data. This lack of a reference standard complicates the evaluation of the sensitivity and specificity of these systems. Increasingly, researchers have compared the detection signals in several sources of surveillance data (for example, syndromic surveillance data for “flu-like illness” with conventional influenza surveillance data). However, the paucity of published data on the sensitivity and specificity of conventional surveillance data prevents a clear understanding of how to interpret the bioterrorism surveillance data. Given the challenges of determining the sensitivity and specificity of a surveillance system from authentic data, surveillance system evaluations based on computer-simulated test data sets of bioterrorism-related outbreaks may provide additional insight into opportunities to improve existing systems. However, this approach will require research on simulation methods for this purpose and standardizing such test data sets (3).
Analyses That Facilitate Public Health Decision Making
Considerable controversy remains about the best methods of data analysis and presentation to facilitate public health decision making based on surveillance data. Most surveillance systems routinely analyze the data by calculating rates of cases over time. Few included reports described the methods for calculating the expected rate of disease or for setting thresholds to determine when the observed rate differs significantly from expected. Several authors described methods for stochastically modeling the spread of communicable disease (206-210). The use of these methods may allow for more accurate determination of the expected rates of disease and deviations from expected. Some of the surveillance systems designed specifically for bioterrorism (for example, ESSENCE) routinely perform both temporal and spatial analyses. The routine application of advanced space–time analytic methods may detect aberrations in bioterrorism surveillance data with greater sensitivity, specificity, and timeliness. However, no published report has evaluated whether a surveillance system that uses both temporal and spatial analyses is probably more timely or sensitive than a system that performs only temporal analyses. We need evaluations of surveillance systems that specifically evaluate various methods of presenting surveillance data to public health officials to determine which methods best facilitate decision making.
Limitations
Our systematic review has 3 potential limitations. First, because the purpose of this project was to synthesize the available evidence on the ability of information technologies to assist clinicians and public health officials during a bioterrorism event, our search strategy and inclusion criteria were designed primarily to collect reports describing information technologies designed for bioterrorism surveillance. We may therefore have neglected to include potentially relevant surveillance systems that use entirely manual methods of collecting bioterrorism surveillance data. Second, many details of the features of the systems were not readily available from the published information about these systems. Although some of the missing information may have been available from the developer or manufacturer of each system, such a survey was outside the scope of this project. Third, data on some existing surveillance systems may not be publicly available. This is probably the case for systems developed by military or public health officials whose objective it is to deploy and maintain surveillance systems for detecting outbreaks in their jurisdiction but whose mandate does not necessarily include publishing those efforts.
Conclusion
Our review identified critical gaps in the literature on the utility of existing surveillance systems to detect illnesses and syndromes potentially related to bioterrorism and highlighted key directions for future evaluations of these systems. Given the striking lack of information on the timeliness, sensitivity and specificity, and ability of systems to facilitate decision making, clinicians and public health officials deploying these systems do so with little scientific evidence to guide them.
Article and Author Information
-
Acknowledgments: The authors thank Emilee Wilhelm and Vandana Sundaram for their assistance preparing this manuscript. They also recognize the contributions of the Stanford University research librarians who helped them design their search strategies: Rikke Greenwald (Lane Medical Library), Ann Latta (Social Sciences Resource Center), Joan Loftus (U.S. Government Documents Bibliographer), and Michael Newman (Falconer Biology Library).
-
Grant Support: This work was performed by the University of California, San Francisco–Stanford Evidence-based Practice Center under contract to the Agency for Healthcare Research and Quality (contract no. 290-97-0013). The project also was supported in part by the U.S. Department of Veterans Affairs.
-
Potential Financial Conflicts of Interest: None disclosed.
-
Requests for Single Reprints: Dena M. Bravata, MD, MS, Center for Primary Care and Outcomes Research, 117 Encina Commons, Stanford, CA 94305-6019; e-mail, bravata{at}healthpolicy.stanford.edu.
-
Current Author Addresses: Drs. Bravata, Haberland, and Owens and Ms. McDonald, Ms. Smith, and Ms. Rydzak: Center for Primary Care and Outcomes Research, 117 Encina Commons, Stanford, CA 94305-6019.
-
Dr. Szeto: Department of Medicine, Kaiser Permanente, 1150 Veterans Boulevard, Redwood City, CA 94063.
-
Dr. Buckeridge: Stanford Medical Informatics, 251 Campus Drive, MSOB X215, Stanford, CA 94305-5479.
References
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.↵
- 6.↵
- 7.↵
- 8.↵
- 9.↵
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.↵
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.↵
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.↵
- 40.↵
- 41.↵
- 42.↵
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.↵
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.↵
- 67.↵
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.↵
- 78.↵
- 79.↵
- 80.↵
- 81.↵
- 82.↵
- 83.↵
- 84.↵
- 85.↵
- 86.↵
- 87.↵
- 88.↵
- 89.↵
- 90.↵
- 91.↵
- 92.↵
- 93.↵
- 94.↵
- 95.↵
- 96.↵
- 97.↵
- 98.↵
- 99.↵
- 100.↵
- 101.↵
- 102.
- 103.
- 104.
- 105.↵
- 106.
- 107.
- 108.↵
- 109.
- 110.
- 111.
- 112.
- 113.
- 114.
- 115.
- 116.
- 117.↵
- 118.
- 119.↵
- 120.↵
- 121.
- 122.
- 123.
- 124.
- 125.
- 126.
- 127.
- 128.
- 129.
- 130.
- 131.
- 132.
- 133.↵
- 134.
- 135.
- 136.
- 137.
- 138.
- 139.↵
- 140.↵
- 141.↵
- 142.↵
- 143.↵
- 144.↵
- 145.↵
- 146.↵
- 147.↵
- 148.↵
- 149.↵
- 150.↵
- 151.↵
- 152.↵
- 153.↵
- 154.↵
- 155.↵
- 156.↵
- 157.↵
- 158.↵
- 159.↵
- 160.↵
- 161.↵
- 162.↵
- 163.↵
- 164.↵
- 165.↵
- 166.↵
- 167.↵
- 168.↵
- 169.↵
- 170.↵
- 171.↵
- 172.↵
- 173.↵
- 174.↵
- 175.↵
- 176.↵
- 177.↵
- 178.↵
- 179.↵
- 180.↵
- 181.↵
- 182.↵
- 183.↵
- 184.↵
- 185.↵
- 186.↵
- 187.↵
- 188.↵
- 189.↵
- 190.↵
- 191.↵
- 192.↵
- 193.↵
- 194.↵
- 195.↵
- 196.↵
- 197.↵
- 198.↵
- 199.↵
- 200.↵
- 201.↵
- 202.↵
- 203.↵
- 204.↵
- 205.↵
- 206.↵
- 207.↵
- 208.↵
- 209.↵
- 210.↵
Related Clinical Content
