Systematic Review: Surveillance Systems for Early Detection of Bioterrorism-Related Diseases

1 June 2004 | Volume 140 Issue 11 | Pages 910-922

Background: Given the threat of bioterrorism and the increasing availability of electronic data for surveillance, surveillance systems for the early detection of illnesses and syndromes potentially related to bioterrorism have proliferated.

Purpose: To critically evaluate the potential utility of existing surveillance systems for illnesses and syndromes related to bioterrorism.

Data Sources: Databases of peer-reviewed articles (for example, MEDLINE for articles published from January 1985 to April 2002) and Web sites of relevant government and nongovernment agencies.

Study Selection: Reports that described or evaluated systems for collecting, analyzing, or presenting surveillance data for bioterrorism-related illnesses or syndromes.

Data Extraction: From each included article, the authors abstracted information about the type of surveillance data collected; method of collection, analysis, and presentation of surveillance data; and outcomes of evaluations of the system.

Data Synthesis: 17 510 article citations and 8088 government and nongovernmental Web sites were reviewed. From these, the authors included 115 systems that collect various surveillance reports, including 9 syndromic surveillance systems, 20 systems collecting bioterrorism detector data, 13 systems collecting influenza-related data, and 23 systems collecting laboratory and antimicrobial resistance data. Only the systems collecting syndromic surveillance data and detection system data were designed, at least in part, for bioterrorism preparedness applications. Syndromic surveillance systems have been deployed for both event-based and continuous bioterrorism surveillance. Few surveillance systems have been comprehensively evaluated. Only 3 systems have had both sensitivity and specificity evaluated.

Limitations: Data from some existing surveillance systems (particularly those developed by the military) may not be publicly available.

Conclusions: Few surveillance systems have been specifically designed for collecting and analyzing data for the early detection of a bioterrorist event. Because current evaluations of surveillance systems for detecting bioterrorism and emerging infections are insufficient to characterize the timeliness or sensitivity and specificity, clinical and public health decision making based on these systems may be compromised.

Key Summary Points The practice of surveillance is changing to address the threat of bioterrorism and to take advantage of the increasing availability of electronic data. The authors identified published descriptions of 29 systems designed specifically for bioterrorism surveillance. Bioterrorism surveillance systems either monitor the incidence of bioterrorism-related syndromes (9) or monitor environmental samples for bioterrorism agents (20). Only 2 syndromic surveillance systems and no environmental monitoring system were evaluated in peer-reviewed studies. Both evaluations of syndromic surveillance systems compared the incidence of flu-like illness syndromes with results from national influenza surveillance. Existing evaluations of surveillance systems for detecting bioterrorism are insufficient to characterize the performance of these systems. Evaluation of bioterrorism surveillance is needed to inform decisions about deploying systems and to facilitate decision making on the basis of system results.

 

The anthrax attacks of 2001 and the recent outbreaks of severe acute respiratory syndrome (SARS) and influenza strikingly demonstrate the continuing threat from illnesses resulting from bioterrorism and related infectious diseases. In particular, these outbreaks have highlighted that an essential component of preparations for illnesses and syndromes potentially related to bioterrorism includes the deployment of surveillance systems that can rapidly detect and monitor the course of an outbreak and thus minimize associated morbidity and mortality (1-3). Driven by the threat of additional outbreaks resulting from bioterrorism and the increasing availability of data available for surveillance, surveillance systems have proliferated. The Centers for Disease Control and Prevention (CDC) defines surveillance systems as those that "collect and analyze morbidity, mortality, and other relevant data and facilitate the timely dissemination of results to appropriate decision makers" (3, 4). However, there is little consensus as to which sources of surveillance data or which collection, analysis, and reporting technologies are probably the most timely, sensitive, and specific for detecting and managing bioterrorism-related illness and related emerging infectious diseases (5).

Existing surveillance systems for bioterrorism-related diseases vary widely with respect to the methods used to collect the surveillance data, surveillance characteristics of the data collected, and analytic methods used to determine when a potential outbreak has occurred. Traditionally, the primary method for collecting surveillance data was manual reporting of suspicious and notifiable clinical and laboratory data from clinicians, hospitals, and laboratories to public health officials (6). Recent innovations in disease surveillance that may improve the timeliness, sensitivity, and specificity of bioterrorism-related outbreak detection include surveillance for syndromes rather than specific diseases and the automated extraction and analysis of routinely collected clinical, administrative, pharmacy, and laboratory data. Little is known about the accuracy of surveillance systems for bioterrorism and related emerging infectious diseases, perhaps because of the diversity of potential data sources for bioterrorism surveillance data; methods for their analysis; and the uncertainty about the costs, benefits, and detection characteristics of each.

Under the auspices of the University of California, San Francisco–Stanford Evidence-based Practice Center, we prepared a comprehensive systematic review that evaluated the ability of available information technologies to inform clinicians and public health officials who are preparing for and responding to bioterrorism and related emerging infectious diseases (7). In this paper, we present the available data on existing systems for surveillance of illnesses and syndromes potentially related to bioterrorism and the published evaluation data on these systems.

Methods

Top Methods Discussion Author & Article Info References

We sought to identify published reports of surveillance systems designed to collect, analyze, and report surveillance data for bioterrorism-related diseases or syndromes or reports of surveillance systems for naturally occurring diseases, if potentially useful for bioterrorism surveillance. We used the U.S. Department of Health and Human Services' definition of bioterrorism-related diseases (8-10). Because most patients with bioterrorism-related diseases initially present with influenza-like illness, acute respiratory distress, gastrointestinal symptoms, febrile hemorrhagic syndromes, and febrile illnesses with either dermatologic or neurologic findings, we considered these conditions to be the bioterrorism-related syndromes. We briefly summarize our methods, which are described in detail elsewhere (7).

Literature Sources and Search Strategies

We searched 3 sources for relevant reports: 5 databases of peer-reviewed articles (for example, MEDLINE, GrayLIT, and National Technical Information Service), government reports, and Web sites of relevant government and commercial entities. We consulted public health, bioterrorism preparedness, and national security experts to identify the 16 government agencies most likely to fund, develop, or use bioterrorism systems (for example, CDC and U.S. Department of Defense). We searched the Web sites of these government agencies and other academic and commercial sites. Finally, we identified additional articles from the bibliographies of included articles and from conference proceedings.

We developed 2 separate search strategies: 1 for MEDLINE (January 1985 to April 2002) and 1 for other sources. In both searches, we included terms such as bioterrorism, biological warfare, information technology, surveillance, public health, and epidemiology. Complete search strategies are available from the authors (7).

Study Selection and Data Abstraction

We reviewed titles, abstracts, and full-length articles to identify potentially relevant articles. Two abstractors, who were blinded to the study authors, abstracted data from all included peer-reviewed articles onto pretested abstraction forms. Given the large volume of Web sites screened, only 1 abstractor, whose work was frequently reviewed by a colleague, collected data from each Web-based report.

Evaluation of Reports of Surveillance Systems

The CDC developed a draft guideline for evaluating public health surveillance systems (3, 11, 12). This guideline recommends that reports of surveillance systems include the following: descriptions of the public health importance of the health event under surveillance; the system under evaluation; the direct costs needed to operate the system; the usefulness of the system; and evaluations of the system's simplicity, flexibility (that is, "the system's ability to change as surveillance needs change"), acceptability ("as reflected by the willingness of participants and stakeholders to contribute to the data collection, analysis and use"), sensitivity to detect outbreaks, positive predictive value of system alarms for true outbreaks, representativeness of the population covered by the system, and timeliness of detection (11, 12). The guideline describes these key elements to consider in an evaluation of a surveillance system but does not provide specific scoring or an evaluation tool. We abstracted information about each CDC criterion from each included reference.

Data Synthesis

We reviewed 17 510 citations of peer-reviewed articles and 8088 Web sites, of which 192 reports on 115 surveillance systems met our inclusion criteria (Figure 1). Of these, 29 systems were designed specifically for detecting bioterrorism-related diseases (as defined by the U.S. Department of Health and Human Services [8-10]) or bioterrorism-related syndromes (for example, flu-like syndrome and fever with rash). An additional 86 systems were designed for surveillance of naturally occurring illnesses, but elements of their design, deployment, or evaluations may be relevant for implementing or evaluating bioterrorism surveillance systems. For example, we included reports of systems for surveillance of nonbiothreat pathogens if they were designed to rapidly transmit surveillance data from sources that could be useful for detecting bioterrorism-related illness (for example, laboratory data, clinicians' reports, hospital-based data, or veterinary data) or if they reported methods of spatial or temporal analyses that facilitated rapid and accurate decision making by public health users. We present the evidence about the systems designed principally for bioterrorism surveillance systems and summarize the evidence about the other surveillance systems.

View larger version (40K): [in this window] [in a new window]   Figure 1. Search results. The literature describing existing systems for illnesses and syndromes potentially related to bioterrorism and the numbers of peer-reviewed evaluations for each category of surveillance systems are presented. The number of references often exceeds the number of surveillance systems because systems were often described in several reports. Also, several reports provided data about systems of more than 1 surveillance type.

 

Surveillance Systems Designed for Bioterrorism-Related Diseases or Syndromes

We identified 2 types of systems for surveillance of bioterrorism-related diseases or syndromes: those that monitor the incidence of bioterrorism-related syndromes and those that collect and transmit bioterrorism detection data from environmental or clinical samples to decision makers.

Surveillance Systems Collecting Syndromic Reports

The 9 surveillance systems designed to monitor the incidence of bioterrorism-related syndromes vary widely with respect to syndromes under surveillance, data collected, flexibility of the data collection tool (for example, some Web-based systems allow remote users to change the prompts given to data collectors), acceptability to data collectors, and methods used to analyze the data (13-23) (Table).

View this table: [in this window] [in a new window]   Table. Surveillance Systems Collecting Syndromic Reports*

 

Two syndromic surveillance systems were evaluated in peer-reviewed reports: the National Health Service Direct system and the program of systematic surveillance of International Classification of Diseases, Ninth Revision (ICD-9), codes from the electronic medical records of the Harvard Vanguard Medical Associates (20, 23). In these evaluations, the numbers of flu-like illnesses or lower respiratory tract syndromes detected by the syndromic surveillance system were similar to the national influenza surveillance data against which they were compared (20, 23). These published evaluation studies lacked information in several key areas: No reports characterized the detection capabilities of syndromic surveillance systems for nonpulmonary syndromes or provided specific information on any of these systems' acceptability, representativeness, or cost. Furthermore, we found no standard definitions for the syndromes under surveillance, and none of the included syndromic surveillance systems that rely on clinicians' entry of patient data defined the syndromes on the collection tool (for example, "flu-like illness" was not defined on the data entry screen or paper tool).

Some other promising systems that were not evaluated in peer-reviewed reports are currently being evaluated (22). These include the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE), which automatically downloads ICD-9 code data from U.S. Department of Defense health care facilities around the world and performs thousands of analyses daily (17). Other local systems, such as the tally sheet system used by the Santa Clara County Public Health Department, collect triage data from emergency department nurses and rely on manual data collection, analysis, and reporting; this information enables syndromic surveillance to occur in settings where electronic medical records are unavailable (22). The Rapid Syndrome Validation Project (RSVP) similarly relies on medically trained staff for collecting surveillance data (21). Physicians enter data on patients presenting with a syndrome of interest into a computer that has a touch-screen interface with RSVP. These systems are being evaluated for various surveillance characteristics, including determination of their sensitivity, specificity, timeliness, and acceptability.

Although most systems for syndromic surveillance are continuously collecting, analyzing, and reporting data, some systems are designed for short-term use at events thought to be potential bioterrorist targets ("event-based" or "drop-in" surveillance). For example, the Lightweight Epidemiology Advanced Detection and Emergency Response System (LEADERS) was used for syndromic surveillance at the 1999 World Trade Organization Summit and the 2001 Presidential Inauguration (19). This system requires staff at participating hospitals to complete a brief Web-based form after each initial patient visit describing the patient's syndrome and whether the patient participated in the event of interest. These syndromic incidence data can be monitored remotely by decision makers. Interpreting surveillance data from event-based surveillance systems can be complicated by the lack of adequate baseline data. For example, if an event-based surveillance system begins collecting surveillance data on 1 or more syndromes of interest a few weeks before the event, pre-event data may be insufficient to calculate an expected rate of cases for the weeks during and immediately after the event of interest. No evaluations of event-based surveillance systems have been published.

Surveillance Systems Collecting Environmental Detection Data

Appendix Table 1 presents the 20 detection systems that transmit data collected from environmental or clinical samples for analysis and presentation to remotely located decision makers. These systems differ in the type and location of sample collected (for example, aerosol samples continuously taken from locations in fixed sites, such as airports or public buildings; environmental samples taken from a site thought to be contaminated by a suspicious powder or other potential bioterrorism exposure; or clinical samples taken from potentially contaminated food, animals, or humans). These systems also differ in the specific technologies used to analyze the samples and send results to data warehouses for analysis and reporting. For example, The Interim Biological Agent Detector is used on U.S. naval ships to continuously monitor the air for a significant increase in particulate concentrations (32, 39-42). If a peak increase is detected, the instrument automatically collects an aerosol sample and alerts the ship's damage control center so the crew can collect and screen the sample with a handheld antigen test. Similar to this naval system, many detection systems were designed by the military and are now being adapted for civilian use. No peer-reviewed evaluations have described these systems; most were described only in government reports and Web-based information provided by manufacturers. None of these reports specifically described timeliness, necessary training, or security measures for specimens or surveillance data.

View this table: [in this window] [in a new window]   Appendix Table 1. Surveillance Systems That Collect or Transmit Bioterrorism Detection Data*

 

Surveillance Systems Designed for Other Purposes

Appendix Table 2 presents the 86 surveillance systems that were not designed for bioterrorism but are potentially relevant for bioterrorism surveillance. Each system is described in detail elsewhere (7). In this paper, we present general information about the types of systems, the evaluation data available about them, and their potential utility for bioterrorism surveillance.

View this table: [in this window] [in a new window]   Appendix Table 2. Systems Collecting Potentially Bioterrorism-Related Surveillance Data

 

Surveillance Systems Collecting Clinical Reports

The 6 surveillance systems that collect clinical information from networks of sentinel clinicians differ with respect to the diseases under surveillance, the frequency and method of reporting, the types of clinicians collecting data, and the timeliness of feedback to clinicians and health departments (55-79). Two of these systems—the French Communicable Disease Network and Eurosentinel—have been described in peer-reviewed evaluation reports (64, 65, 80). A retrospective evaluation of the French Communicable Disease Network found that the combination of clinicians' reports with information on viral isolates from the French Reference Centers was more timely than surveillance performed with viral isolates alone (65). The Eurosentinel project uses an international group of volunteer physicians who submit weekly reports to a coordinating center in Belgium. Outputs for influenza are available within minutes of reporting; however, data for other diseases are released in a quarterly newsletter (57). A report describing the first 3 years of the project found that discrepancies in disease-reporting practices, particularly the use of different denominators among the sentinel networks from different countries, made it difficult to compare the data from participating networks (57).

Surveillance Systems Collecting Influenza-related Data

Our search identified 13 systems for influenza surveillance (15, 81-101), of which 5 systems have been described in peer-reviewed evaluation reports (84, 87, 88, 90, 97). In general, these evaluations indicate that electronic reporting methods are more timely than manual systems (87, 88).

There is no clear consensus as to the most sensitive, specific, or timely data for influenza surveillance. An analysis of data from the Regional Influenza Surveillance Group of France found that sick-leave prescriptions, emergency house calls, and numbers of patients with influenza-like illness seen by general practitioners and pediatricians were the most sensitive indicators for the early recognition of influenza (87, 88). In contrast, data from the Viral Watch Program of South Africa suggest that viral isolates are more sensitive indicators of influenza activity than school absenteeism or mortality rates (84). A comparison of school absenteeism data collected by the Japanese School Health Surveillance System with data from the national influenza surveillance system demonstrated a sensitivity of 80% and a specificity of 100% (90). However, the authors noted that gaps in surveillance data during school holidays and the possible inclusion of non–influenza virus infections (for example, adenovirus) complicated the use of these data for influenza surveillance (90). These results do not provide sufficient evidence to favor the use of any given source of influenza data or method of collection or analysis.

Surveillance Systems Collecting Laboratory Data

Evaluations of systems for the surveillance of laboratory and antimicrobial resistance suggest that automated laboratory reporting systems are generally more timely and sensitive than conventional reporting methods (108, 117, 119, 120, 133). The sensitivity of these systems (typically compared with manual systems) ranged from 76% to 100% (117, 120); the specificity (95%) was reported for only 1 system (117). Few reports described methods for manipulating samples or confirming results, acceptability, or cost. No system was evaluated specifically for detecting a biothreat agent.

Surveillance Systems Collecting Foodborne Illness Data

We found 7 systems that collect and analyze reports from clinicians or laboratories about the incidence and characteristics of foodborne pathogens (139-150) and 3 systems that model microbial growth responses to food production methods (151-154). Evaluation data on these systems are limited to estimates of disease incidence identified by the systems but do not further describe the systems' sensitivity, specificity, or timeliness.

Surveillance Systems Collecting Zoonotic and Animal Disease Data

We found 2 systems for surveillance of zoonotic illnesses and 4 systems for the surveillance of animal diseases (155-166). None has been described in a peer-reviewed evaluation. Most reports provide little or no information about the timeliness of these systems; those that did suggest lag times are too long for effective bioterrorism surveillance.

Surveillance Systems Collecting Other Kinds of Data

We found 16 systems designed specifically for hospital surveillance (167-190). Evaluations of some hospital surveillance systems reported improvements in the timeliness and sensitivity of detecting nosocomial infections when compared with manual methods (168-170, 175, 176, 178, 183). An additional 12 systems met our inclusion criteria but did not belong in the preceding categorizations, including 6 systems that collect data about specific groups of patients (81, 86, 105, 191-196), 2 systems that collect pharmacy data (197, 198), and other systems (199-203). Evaluations of these systems generally showed little evidence that these systems have sufficient sensitivity, specificity, or timeliness to detect a bioterrorist event.

Evaluation of Reports of Surveillance Systems

When applying the CDC's guidelines for evaluating reports of surveillance systems, we abstracted whether the authors specifically described each characteristic of interest (Figure 2). The discussion of these characteristics was often modest and was based on opinion rather than formal evaluation (for example, some authors reported that the system under evaluation "was sensitive" without reporting actual sensitivity or specificity). Only 1 report addressed all 9 CDC criteria (90). Seventy-two reports of 43 systems described their timeliness, 29 reports of 22 systems described their sensitivity, and 15 reports of 12 systems described their specificity; however, only 12 reports of 9 systems described all 3 characteristics. Only 3 reports of 3 systems provided numeric data for both sensitivity and specificity of the system (90, 117, 175).

View larger version (24K): [in this window] [in a new window]   Figure 2. Application of the Centers for Disease Control and Prevention evaluation guideline to peer-reviewed reports of surveillance systems.

 

Discussion

Top Methods Discussion Author & Article Info References

Our systematic review identified 115 existing surveillance systems, 29 of which were designed for surveillance of illnesses and syndromes associated with bioterrorism-relevant pathogens. The evidence used to judge the usefulness of the reviewed systems is limited. Of the studies that evaluated systems for their intended purpose, few adhered to the CDC's published criteria for high-quality evaluations of surveillance systems. Even if a system was found useful for its intended purpose (for example, surveillance for influenza), we can only infer that the system might be useful for responding to bioterrorism.

Systems for bioterrorism surveillance require 3 key features: timeliness, high sensitivity and specificity, and routine analysis and presentation of the data that facilitate public health decision making. We discuss each characteristic in the following sections.

Timeliness

Effective surveillance for bioterrorism-related illness depends on systems that promptly collect, analyze, and report data to decision makers, because the effectiveness of intervention after a bioterrorism attack has been strongly linked to the rapidity of detection (204, 205). The evaluations of surveillance systems demonstrated 2 key factors affecting their timeliness. First, in general, the electronic collection and reporting of surveillance data improved detection compared with older, manual methods. Despite the advantages of electronic collection and reporting and the increasing availability of administrative and medical record data that can be transmitted instantaneously, many local health departments do not currently have adequate resources to manage, analyze, and interpret such large data sets. Also, as the size and complexity of the data under surveillance increase, so does the time required to analyze and interpret the data. Some systems that facilitate manual reporting of suspicious cases by clinicians and triage staff through fax or computer entry to public health officials may substantially reduce delays in reporting and represent programs that could be used in places without electronic medical records or electronic disease reporting or in health departments without extensive electronic data management resources. Surveillances systems must be evaluated to specifically delineate the time required for each step in the surveillance process from initial data collection to arrival of data at the health department to decision making about outbreak investigation.

Second, the timeliness of a surveillance system is affected by the source of surveillance data. For example, school and work absenteeism, calls to telephone care nurses, and over-the-counter pharmacy sales may provide earlier indications of bioterrorism than hospital discharge data or coroners' reports. Relatively few of the 115 included systems collect the earliest types of surveillance data—a potentially important gap in available surveillance systems. Systems that collect pharmaceutical data, such as EPIFAR (198), are promising for bioterrorism surveillance. Pharmaceutical data, particularly over-the-counter medication sales data, can indicate an outbreak, although these data would probably not be specific for bioterrorism. In addition, most pharmaceutical sales are tracked electronically. The detection characteristics of common prescription and nonprescription medications used for bioterrorism-related syndromes must be carefully analyzed to determine the utility of these data for bioterrorism surveillance. Similarly, surveillance systems must be evaluated to compare the timeliness of detection on the basis of the source of data used. Evaluations that determine how integration of several data sources affects the timeliness and accuracy of the system are also needed.

Sensitivity and Specificity

Bioterrorism surveillance systems with inadequate sensitivity may fail to detect cases of bioterrorism-related illness, which could result in substantial delays in detection and potentially catastrophic increases in morbidity and mortality. Systems with inadequate specificity may have frequent false alarms, which may result in costly actions by clinicians and public health officials or, perhaps even worse, officials ignoring the system when it reports a suspicious event. Because sensitivity and specificity are related, they must be evaluated simultaneously. However, only 3 reports of 3 systems provided numeric data for both sensitivity and specificity of the system (90, 117, 175). This substantially limits our understanding of the accuracy of existing surveillance systems for bioterrorism-related illness.

In addition, because there have been so few cases of bioterrorism-related illness, there are no reference standards against which to compare the surveillance data. This lack of a reference standard complicates the evaluation of the sensitivity and specificity of these systems. Increasingly, researchers have compared the detection signals in several sources of surveillance data (for example, syndromic surveillance data for "flu-like illness" with conventional influenza surveillance data). However, the paucity of published data on the sensitivity and specificity of conventional surveillance data prevents a clear understanding of how to interpret the bioterrorism surveillance data. Given the challenges of determining the sensitivity and specificity of a surveillance system from authentic data, surveillance system evaluations based on computer-simulated test data sets of bioterrorism-related outbreaks may provide additional insight into opportunities to improve existing systems. However, this approach will require research on simulation methods for this purpose and standardizing such test data sets (3).

Analyses That Facilitate Public Health Decision Making

Considerable controversy remains about the best methods of data analysis and presentation to facilitate public health decision making based on surveillance data. Most surveillance systems routinely analyze the data by calculating rates of cases over time. Few included reports described the methods for calculating the expected rate of disease or for setting thresholds to determine when the observed rate differs significantly from expected. Several authors described methods for stochastically modeling the spread of communicable disease (206-210). The use of these methods may allow for more accurate determination of the expected rates of disease and deviations from expected. Some of the surveillance systems designed specifically for bioterrorism (for example, ESSENCE) routinely perform both temporal and spatial analyses. The routine application of advanced space–time analytic methods may detect aberrations in bioterrorism surveillance data with greater sensitivity, specificity, and timeliness. However, no published report has evaluated whether a surveillance system that uses both temporal and spatial analyses is probably more timely or sensitive than a system that performs only temporal analyses. We need evaluations of surveillance systems that specifically evaluate various methods of presenting surveillance data to public health officials to determine which methods best facilitate decision making.

Limitations

Our systematic review has 3 potential limitations. First, because the purpose of this project was to synthesize the available evidence on the ability of information technologies to assist clinicians and public health officials during a bioterrorism event, our search strategy and inclusion criteria were designed primarily to collect reports describing information technologies designed for bioterrorism surveillance. We may therefore have neglected to include potentially relevant surveillance systems that use entirely manual methods of collecting bioterrorism surveillance data. Second, many details of the features of the systems were not readily available from the published information about these systems. Although some of the missing information may have been available from the developer or manufacturer of each system, such a survey was outside the scope of this project. Third, data on some existing surveillance systems may not be publicly available. This is probably the case for systems developed by military or public health officials whose objective it is to deploy and maintain surveillance systems for detecting outbreaks in their jurisdiction but whose mandate does not necessarily include publishing those efforts.

Conclusion

Our review identified critical gaps in the literature on the utility of existing surveillance systems to detect illnesses and syndromes potentially related to bioterrorism and highlighted key directions for future evaluations of these systems. Given the striking lack of information on the timeliness, sensitivity and specificity, and ability of systems to facilitate decision making, clinicians and public health officials deploying these systems do so with little scientific evidence to guide them.

Author and Article Information

Top Methods Discussion Author & Article Info References

From the University of California, San Francisco–Stanford Evidence-based Practice Center and Center for Primary Care and Outcomes Research, Stanford University School of Medicine, Stanford, California; Veterans Affairs Palo Alto Healthcare System, Palo Alto, California; and Kaiser Permanente, Redwood City, California.

Acknowledgments: The authors thank Emilee Wilhelm and Vandana Sundaram for their assistance preparing this manuscript. They also recognize the contributions of the Stanford University research librarians who helped them design their search strategies: Rikke Greenwald (Lane Medical Library), Ann Latta (Social Sciences Resource Center), Joan Loftus (U.S. Government Documents Bibliographer), and Michael Newman (Falconer Biology Library).

Grant Support: This work was performed by the University of California, San Francisco–Stanford Evidence-based Practice Center under contract to the Agency for Healthcare Research and Quality (contract no. 290-97-0013). The project also was supported in part by the U.S. Department of Veterans Affairs.

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: Dena M. Bravata, MD, MS, Center for Primary Care and Outcomes Research, 117 Encina Commons, Stanford, CA 94305-6019; e-mail, bravata{at}healthpolicy.stanford.edu.

Current Author Addresses: Drs. Bravata, Haberland, and Owens and Ms. McDonald, Ms. Smith, and Ms. Rydzak: Center for Primary Care and Outcomes Research, 117 Encina Commons, Stanford, CA 94305-6019.

Dr. Szeto: Department of Medicine, Kaiser Permanente, 1150 Veterans Boulevard, Redwood City, CA 94063.

Dr. Buckeridge: Stanford Medical Informatics, 251 Campus Drive, MSOB X215, Stanford, CA 94305-5479.

References

Top Methods Discussion Author & Article Info References

1.  Koplan J. CDC's strategic plan for bioterrorism preparedness and response. Public Health Rep. 2001;116(Suppl 2):9-16. [PMID: 11880662].[Medline]

2.  The operational response to SARS. Geneva: World Health Organization; 2003. Accessed at http://www.who.int/csr/sars/goarn2003_4_16/en/ on 12 May 2003.

3.  Sosin DM. Draft framework for evaluating syndromic surveillance systems. J Urban Health. 2003;80:i8-13. [PMID: 12791773].[Medline]

4.  Langmuir AD. The surveillance of communicable diseases of national importance. N Engl J Med. 1963;268:182-92. [PMID: 13928666].

5.  Lober WB, Karras BT, Wagner MM, Overhage JM, Davidson AJ, Fraser H, et al. Roundtable on bioterrorism detection: information system-based surveillance. J Am Med Inform Assoc. 2002;9:105-15. [PMID: 11861622].[Abstract/Free Full Text]

6.  Teutsch SM, Churchill RE, eds. Principles and Practice of Public Health Surveillance. New York: Oxford Univ Pr; 1994.

7.  Bravata DM, McDonald K, Owens DK, Buckeridge, D, Haberland C, Rydzak C, et al. Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems. Evidence Report/Technology Assessment no. 59. Prepared by the University of California, San Francisco–Stanford Evidence-based Practice Center for the Agency for Healthcare Research and Quality. Rockville, MD: Agency for Healthcare Research and Quality; 2002. Accessed at http://www.ahrq.gov/clinic/bioitinv.htm on 1 April 2004.

8.  Darling RG, Catlett CL, Huebner KD, Jarrett DG. Threats in bioterrorism. I: CDC category A agents. Emerg Med Clin North Am. 2002;20:273-309. [PMID: 12120480].[Medline]

9.  Moran GJ. Threats in bioterrorism. II: CDC category B and C agents. Emerg Med Clin North Am. 2002;20:311-30. [PMID: 12120481].[Medline]

10.  Rotz LD, Khan AS, Lillibridge SR, Ostroff SM, Hughes JM. Public health assessment of potential biological terrorism agents. Emerg Infect Dis. 2002;8:225-30. [PMID: 11897082].[Medline]

11.  Guidelines for evaluating surveillance systems. MMWR Morb Mortal Wkly Rep. 1988;37(Suppl 5):1-18. [PMID: 3131659].

12.  Sosin DM. Draft Framework for Evaluating Syndromic Surveillance Systems for Bioterrorism Preparedness. Atlanta, GA: Centers for Disease Control and Prevention; 2003. Accessed at http://www.cdc.gov/epo/dphsi/syndromic.htm on 27 February 2004.

13.  Surveillance systems: home pages and contacts. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/osr/survsyss.htm on 21 September 2001. Now available at http://www.cdc.gov/ncidod/osr/site/surv_resources/surv_sys.htm.

14.  Doyle TJ, Bryan RT. Infectious disease morbidity in the US region bordering Mexico, 1990-1998. J Infect Dis. 2000;182:1503-10. [PMID: 11010841].[Medline]

15.  Global Emerging Infections System: Annual Report Fiscal Year 1999. Silver Spring, MD: U.S. Department of Defense; 1999.

16.  Corwin A. Developing regional outbreak response capabilities: Early Warning Outbreak Recognition System (EWORS). Navy Med. 2000;Sept/Oct:1-5.

17.  Pavlin JA, Kelley PW. ESSENCE: Electronic Surveillance System for the Early Notification of Community-based Epidemics. Silver Spring, MD: U.S. Department of Defense, Global Emerging Infections Surveillance and Response System; 4 October 2001.

18.  Health Buddy System. Mountain View, CA: Health Hero Network. Accessed at http://www.healthhero.com on 10 November 2001.

19.  Idaho Technology products. Salt Lake City, UT: Idaho Technology, Inc. Accessed at http://www.idahotech.com on 29 October 2001.

20.  Harcourt SE, Smith GE, Hollyoak V, Joseph CA, Chaloner R, Rehman Y, et al. Can calls to NHS Direct be used for syndromic surveillance? Commun Dis Public Health. 2001;4:178-82. [PMID: 11732356].[Medline]

21.  Zelicoff A, Brillman J, Forslund DW, George JE, Zink S, Koenig S, et al. The Rapid Syndrome Validation Project (RSVP). Albuquerque, NM: Sandia National Laboratories; 2001.

22.  Bravata DM, Rahman MM, Luong N, Divan HA, Cody SH. A comparison of syndromic incidence data collected by triage nurses in Santa Clara County with regional infectious disease data [Abstract]. J Urban Health. 2003;80:122.

23.  Lazarus R, Kleinman KP, Dashevsky I, DeMaria A, Platt R. Using automated medical records for rapid identification of illness syndromes (syndromic surveillance): the example of lower respiratory infection. BMC Public Health. 2001;1:9 [PMID: 11722798].[Medline]

24.  Automated Decision Aid System for Hazardous Incidents (ADASHI). Aberdeen Proving Ground. Accessed at http://www.apg.army.mil on 21 August 2001. Now available at http://www.adashi.org.

25.  Non-EPA databases and software. Chemical Emergency Preparedness and Prevention Office, Office of Solid Waste and Emergency Response, U.S. Environmental Protection Agency. Accessed at http://www.epa.gov/ceppo/ds-noep.htm#midas-at/ on 27 October 2001. Now available at http://yosemite.epa.gov/oswer/ceppoweb.nsf/content/ds-noep.htm.

26.  Meterological Information and Dispersion Assessment System Anti-Terrorism (MIDAS-AT). Rockville, MD: ABS Consulting. Accessed at http://www.midas-at.com/plg-home.html on 27 September 2001. Now available at http://www.absconsulting.com/midas/index.html.

27.  Bruhn NewTech products. Søborg, Denmark: Bruhn NewTech. Accessed at http://www.newtech.dk/nbc-products.htm on 28 September 2001.

28.  NBC Command and Control. NBC Industry Group. Accessed at http://www.nbcindustrygroup.com/handbook/pdf/COMMUNICATIONS.pdf on 28 September 2001.

29.  Real-Time: Emergency Response System for Fixed Facilities. Camarillo, CA: SAFER Systems, LLC; 2001. Accessed at http://www.safersystem.com/RealTime.htm on 28 September 2001.

30.  Gensor, Inc. Accessed at http://www.gensor.com on 16 October 2001.

31.  The AMEBA Biosensor. Huntingdon Valley, PA: Gensor, Inc. Accessed at http://www.gensor.com/pages/267478/index.htm?gen_time=1033067403933 on 1 April 2004.

32.  Biological Detection System Technologies Technology and Industrial Base Study: A Primer on Biological Detection Technologies. Fort Belvoir, VA: North American Technology and Industrial Base Organization; February 2001. Accessed at http://www.dtic.mil/natibo/docs/BioDetectReport-2.pdf on 21 April 2004.

33.  RAPTOR Monroe, WA: Research International, Inc. Accessed at http://www.resrchintl.com/images/raptor_fs_report_010300.pdf on 5 October 2001. Now available at http://www.resrchintl.com/raptor.html.

34.  Chem-bio products. Research International, Inc. Accessed at http://www.nbcindustrygroup.com/reseinte.htm on 28 September 2001.

35.  Research International, Inc. NBC Industry Group. Accessed at http://www.nbcindustrygroup.com/reseinte.htm on 5 October 2001. Now available at http://www.resrchintl.com.

36.  Analyte 2000: Biosensor breakthrough! FoodTech Source. Accessed at http://www.foodtechsource.com/emag/017/gadgets.htm on 5 October 2001.

37.  Analyte 2000: Biological detection. Monroe, WA: Research International, Inc. Accessed at http://www.resrchintl.com/analyte2000.html on 1 April 2004.

38.  King KD, Anderson GP, Bullock KE, Regina MJ, Saaski EW, Ligler FS. Detecting staphylococcal enterotoxin B using an automated fiber optic biosensor. Biosens Bioelectron. 1999;14:163-70. [PMID: 10101838].[Medline]

39.  Campbell J, Francesconi S, Boyd J, Worth L, Moshier T. Environmental air sampling to detect biological warfare agents. Mil Med. 1999;164:541-2. [PMID: 10459261].[Medline]

40.  U.S. Department of Defense. Chemical and Biological Defense Program: Annual Report to Congress. Washington, DC: U.S. Department of Defense; March 2000.

41.  Communicable disease surveillance and response. World Health Organization/OMS. 24 April 1998. Accessed at http://www.who.int/emc/surveill/index1.html on 16 May 2001. Now available at http://www.who.int/csr/en/.

42.  Chemical and Biological Terrorism: Research and Development to Improve Civilian Medical Response. Washington, DC: National Academy Pr; 1999.

43.  Fluorescence Aerodynamic Particle Sizer (FLAPS). Defence Research and Development Canada–Suffield (formerly Defense Research Establishment Suffield). Accessed at http://www.dres.dnd.ca/products/RD98001/index.html on 9 September 2001. Now available at http://www.suffield.drdc-rddc.gc.ca/ResearchTech/Products/CB_PRODUCTS/RD98001/index_e.html.

44.  Preliminary product information: Model 3312 Ultraviolet Aerodynamic Particle Sizer Spectrometer. Shoreview, MN: TSI Inc. Accessed at http://www.tsi.com/particle/products/3312a/3312aint.html on 5 October 2001. Now available at http://www.tsi.com/particle/products/partsize/3321.htm.

45.  Emergency preparedness: PROTECT. Argonne, IL: Argonne National Laboratory. Accessed at http://www.dis.anl.gov/ep/PROTECT/ep_PROTECT_home.html on 23 September 2001.

46.  Emergency preparedness: PROTECT fact sheet. Argonne, IL: Argonne National Laboratory. Accessed at http://www.dis.anl.gov/ep/PROTECT/ep_PROTECT_factsheet.html on 15 August 2001.

47.  Suliga W. Short Range Biological Standoff Detection System (SR-BSDS). Herndon, VA: Fibertek, Inc. Accessed at http://www.cbwsymp.foa.se/abstr/New_Equipment/suliga.pdf on 5 October 2001.

48.  Weitekamp M, McCready P. The Biological Aerosol Sentry and Information System (BASIS). Livermore, CA: Lawrence Livermore National Laboratory. Accessed at http://greengenes.llnl.gov/bbrp/html/mccreadyabst.html on 1 March 2002.

49.  Joint Biological Point Detection System (JBPDS). U.S. Departments of Defense, Transportation, and Energy. Accessed at http://www.dote.osd.mil/reports/FY00/other/00jbpds.html on 25 October 2001. Now available at http://hqinet001.hqmc.usmc.mil/p&r/concepts/2003/PDF/Chapter%204%20C&P%20PDFs/ch4%20p4%20CSSE%20JBPDS.pdf.

50.  Maritime Systems and Sensors. Lockheed Martin. Accessed at http://www.lockheedmartin.com/manassas/ on 28 October 2002.

51.  4WARN. Ottawa, Ontario, Canada: General Dynamics Canada (formerly Computing Devices Canada). Accessed at http://www.computingdevices.com/land/4warn/index.htm on 29 October 2001.

52.  Reducing the threat of biological weapons. Science & Technology Review. 1998;June:4-9. Accessed at http://www.llnl.gov/str/Milan.html on 7 September 2001.

53.  Mobile Atmospheric Sampling and Identification Facility. Defence Research and Development Canada–Suffield (formerly Defense Research Establishment Suffield). Accessed at http://www.dres.dnd.ca/Products/RD95006/index.html on 26 September 2001. Now available at http://www.suffield.drdc-rddc.gc.ca/ResearchTech/Products/CB_PRODUCTS/RD95006/index_e.html.

54.  NBC Product and Services Handbook. The U.S. Joint Service Material Group (JSMG) and the NBC Industry Group; 2001. Accessed at http://www.nbcindustrygroup.com/handbook/pdf/COMMUNICATIONS.pdf on 1 October 2001.

55.  Talan DA, Moran GJ, Mower WR, Newdow M, Ong S, Slutsker L, et al. EMERGEncy ID NET: an emergency department-based emerging infections sentinel network. Clin Infect Dis. 1999;28:401-2. [PMID: 10064261].[Medline]

56.  Talan DA, Moran GJ, Mower WR, Newdow M, Ong S, Slutsker L, et al. EMERGEncy ID NET: an emergency department-based emerging infections sentinel network. The EMERGEncy ID NET Study Group. Ann Emerg Med. 1998;32:703-11. [PMID: 9832668].[Medline]

57.  van Casteren V, Leurquin P. Eurosentinel: development of an international sentinel network of general practitioners. Methods Inf Med. 1992;31:147-52. [PMID: 1635466].[Medline]

58.  Surveillance of tuberculosis in Europe. EuroTB. Accessed at http://www.eurotb.org on 29 March 2004.

59.  EuroTB (InVS/KNCV) and the national coordinators for tuberculosis surveillance in the WHO European Region. Surveillance of tuberculosis in Europe. Report on tuberculosis cases notified in 2000. March 2003. Accessed at http://www.eurotb.org/index.htm on 21 April 2004.

60.  Schwoebel V, Antoine D, Veen J. Surveillance of tuberculosis in Europe. Med Arh. 1999;53:9-10. [PMID: 10546459].[Medline]

61.  Influenza-like diseases. Surveillance of influenza-like diseases through a national computer network, 1984-1989. Wkly Epidemiol Rec. 1990;65:103-4. [PMID: 2386673].[Medline]

62.  Garnerin P, Valleron AJ. The French communicable diseases computer network: a technical view. Comput Biol Med. 1992;22:189-200. [PMID: 1617953].[Medline]

63.  Flahault A, Garnerin P, Chauvin P, Farran N, Saidi Y, Diaz C, et al. Sentinelle traces of an epidemic of acute gastroenteritis in France. Lancet. 1995;346:162-3. [PMID: 7603234].[Medline]

64.  Carrat F, Flahault A, Boussard E, Farran N, Dangoumau L, Valleron AJ. Surveillance of influenza-like illness in France. The example of the 1995/1996 epidemic. J Epidemiol Community Health. 1998;52(Suppl 1):32S-38S. [PMID: 9764269].

65.  Costagliola D, Flahault A, Galinec D, Garnerin P, Menares J, Valleron AJ. A routine tool for detection and assessment of epidemics of influenza-like syndromes in France. Am J Public Health. 1991;81:97-9. [PMID: 1983924].[Abstract/Free Full Text]

66.  Garnerin P, Saidi Y, Valleron AJ. The French Communicable Diseases Computer Network. A seven-year experiment. Ann N Y Acad Sci. 1992;670:29-42. [PMID: 1309100].[Medline]

67.  Parsons DF, Garnerin P, Flahault A, Gotham IJ. Status of electronic reporting of notifiable conditions in the United States and Europe. Telemed J. 1996;2:273-84. [PMID: 10165364].[Medline]

68.  Valleron AJ, Bouvet E, Garnerin P, Menares J, Heard I, Letrait S, et al. A computer network for the surveillance of communicable diseases: the French experiment. Am J Public Health. 1986;76:1289-92. [PMID: 3766824].[Abstract/Free Full Text]

69.  Chauvin P, Valleron AJ. Monitoring the compliance of sentinel general practitioners in public health surveillance: which GPs persevere? Int J Epidemiol. 1997;26:166-72. [PMID: 9126517].[Abstract/Free Full Text]

70.  Valleron AJ, Garnerin P. Computerised surveillance of communicable diseases in France. Commun Dis Rep CDR Rev. 1993;3:R82-7. [PMID: 7693158].

71.  Massari V, Maison P, Desenclos JC, Flahault A. Six years of sentinel surveillance of hepatitis B in general practice in France. Eur J Epidemiol. 1998;14:765-7. [PMID: 9928870].[Medline]

72.  Boussard E, Flahault A, Vibert JF, Valleron AJ. Sentiweb: French communicable disease surveillance on the World Wide Web. BMJ. 1996;313:1381-2; discussion 1382-4. [PMID: 8956709].[Free Full Text]

73.  SentiWeb. Le Réseau Sentinelles. Accessed at http://www.b3e.jussieu.fr/sentiweb/ on 2 October 2001.

74.  GeoSentinel: Surveillance strategy. The Global Surveillance Network of the International Society of Travel Medicine and Centers for Disease Control and Prevention. Accessed at http://www.istm.org/geosentinel/surveill.html on 21 September 2001.

75.  GeoSentinel: Objectives. The Global Surveillance Network of the International Society of Travel Medicine and Centers for Disease Control and Prevention. Accessed at http://www.istm.org/geosentinel/objectiv.html on 21 September 2001.

76.  National Electronic Telecommunications System for Surveillance—United States, 1990-1991. MMWR Morb Mortal Wkly Rep. 1991;40:502-3. [PMID: 1649378].[Medline]

77.  Reporting race and ethnicity data—National Electronic Telecommunications System for Surveillance, 1994-1997. MMWR Morb Mortal Wkly Rep. 1999;48:305-12. [PMID: 10227798].[Medline]

78.  Birkhead G, Chorba TL, Root S, Klaucke DN, Gibbs NJ. Timeliness of national reporting of communicable diseases: the experience of the National Electronic Telecommunications System for Surveillance. Am J Public Health. 1991;81:1313-5. [PMID: 1928531].[Abstract/Free Full Text]

79.  Graitcer PL, Burton AH. The Epidemiologic Surveillance Project: a computer-based system for disease surveillance. Am J Prev Med. 1987;3:123-7. [PMID: 2838060].[Medline]

80.  Surveillance of influenza-like diseases through a national computer network—France, 1984-1989. MMWR Morb Mortal Wkly Rep. 1989;38:855-7. [PMID: 2511415].[Medline]

81.  Heymann DL, Rodier GR. Global surveillance of communicable diseases. Emerg Infect Dis. 1998;4:362-5. [PMID: 9716946].[Medline]

82.  Williams RJ, Cox NJ, Regnery HL, Noah DL, Khan AS, Miller JM, et al. Meeting the challenge of emerging pathogens: the role of the United States Air Force in global influenza surveillance. Mil Med. 1997;162:82-6. [PMID: 9038023].[Medline]

83.  Flahault A, Dias-Ferrao V, Chaberty P, Esteves K, Valleron AJ, Lavanchy D. FluNet as a tool for global monitoring of influenza on the Web. JAMA. 1998;280:1330-2. [PMID: 9794312].[Abstract/Free Full Text]

84.  Schoub BD, Johnson S, McAnerney J, Blackburn NK. Benefits and limitations of the Witwatersrand influenza and acute respiratory infections surveillance programme. S Afr Med J. 1994;84:674-8. [PMID: 7839255].[Medline]

85.  Overview of influenza surveillance. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/diseases/flu/flusurv.htm on 24 September 2001. Now available at http://www.cdc.gov/flu/weekly/fluactivity.htm.

86.  Surveillance resources. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/osr/internetsurvsyss.htm on 24 September 2001. Now available at http://www.cdc.gov/ncidod/osr/site/surv_resources/index.htm.

87.  Hannoun C, Dab W, Cohen JM. A new influenza surveillance system in France: the Ile-de-France "GROG". 1. Principles and methodology. Eur J Epidemiol. 1989;5:285-93. [PMID: 2792306].[Medline]

88.  Quenel P, Dab W. Influenza A and B epidemic criteria based on time-series analysis of health services surveillance data. Eur J Epidemiol. 1998;14:275-85. [PMID: 9663521].[Medline]

89.  National Flu Surveillance Network (NFSN). Accessed at http://www.fluwatch.com/index2.html on 27 November 2001.

90.  Takahashi H, Fujii H, Shindo N, Taniguchi K. Evaluation of the Japanese school health surveillance system for influenza. Jpn J Infect Dis. 2001;54:27-30. [PMID: 11326126].[Medline]

91.  Snacken R, Manuguerra JC, Taylor P. European Influenza Surveillance Scheme on the Internet. Methods Inf Med. 1998;37:266-70. [PMID: 9787627].[Medline]

92.  Fleming DM, Cohen JM. Experience of European collaboration in influenza surveillance in the winter of 1993-1994. J Public Health Med. 1996;18:133-42. [PMID: 8816310].[Abstract/Free Full Text]

93.  Snacken R, Bensadon M, Strauss A. The CARE telematics network for the surveillance of influenza in Europe. Methods Inf Med. 1995;34:518-22. [PMID: 8713768].[Medline]

94.  Sprenger MJ, Kempen BM, Hannoun C, Masurel N. Electronic influenza surveillance [Letter]. Lancet. 1992;339:874 [PMID: 1347885].[Medline]

95.  Canas LC, Lohman K, Pavlin JA, Endy T, Singh DL, Pandey P, et al. The Department of Defense laboratory-based global influenza surveillance system. Mil Med. 2000;165:52-6. [PMID: 10920641].[Medline]

96.  The DoD Worldwide Influenza Surveillance Program. U.S. Department of Defense Global Emerging Infections System. Accessed at http://www.geis.ha.osd.mil/getpage.asp?page=flusurv.htm&action=7&click=KeyProgram on 14 May 2001. Now available at http://www.geis.ha.osd.mil/GEIS/SurveillanceActivities/Influenza/influenza.asp.

97.  Johnson N, Mant D, Jones L, Randall T. Use of computerised general practice data for population surveillance: comparative study of influenza data. BMJ. 1991;302:763-5. [PMID: 2021767].

98.  121 Cities Mortality Reporting System: History. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/epo/dphsi/121hist.htm on 21 September 2001.

99.  121 Cities Mortality Reporting System: Use of the data. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/epo/dphsi/121dtuse.htm on 5 September 2001.

100.  Szecsenyi J, Uphoff H, Ley S, Brede HD. Influenza surveillance: experiences from establishing a sentinel surveillance system in Germany. J Epidemiol Community Health. 1995;49(Suppl 1):9-13. [PMID: 7561670].

101.  California Influenza Surveillance Project. California Department of Health Services: Viral and Rickettsial Disease Laboratory. Accessed at http://www.dhs.ca.gov/ps/dcdc/VRDL/html/FLU/Fluintro.htm on 2 February 2002.

102.  Active Bacterial Core Surveillance: Know Your ABCs. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/dbmd/abcs/default.htm on 21 September 2001.

103.  Active Bacterial Core Surveillance: Objectives. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/dbmd/abcs/objectives.htm on 21 September 2001.

104.  Active Bacterial Core Surveillance: Surveillance Population. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/dbmd/abcs/survpopulation.htm on 21 September 2001.

105.  State of California: Bioterrorism Surveillance and Epidemiologic Response Plan. Sacramento, CA: Department of Health Services, State of California Health and Human Services Agency; 2002.

106.  Rypka EW, Madar DA. Enhancing laboratory detection of disease. Part 2: Amplifying information and requisite variety. Am Clin Lab. 1997;16:14-7. [PMID: 10172959].[Medline]

107.  Rypka EW, Madar DA. Enhancing laboratory detection of disease. Part 3: Detecting a hantavirus disease event: a case study. Am Clin Lab. 1997;16:6-7. [PMID: 10173024].[Medline]

108.  Effler P, Ching-Lee M, Bogard A, Ieong MC, Nekomoto T, Jernigan D. Statewide system of electronic notifiable disease reporting from clinical laboratories: comparing automated reporting with conventional methods. JAMA. 1999;282:1845-50. [PMID: 10573276].[Abstract/Free Full Text]

109.  National Science and Technology Council Committee on International Science, Engineering and Technology. Emerging Infectious Disease Task Force, Annual Report. Washington, DC: Office of Science and Technology Policy; 1997.

110.  Emerging Pathogens Initiative: an automated surveillance system. Emerg Infect Dis. 1999;5:314.

111.  Kralovic SM, Kelly AA, Danko LH, Simbartl LA. Penicillin-resistant Streptococcus pneumoniae (PRSP) in the Department of Veterans Affairs, a 33 month review [Abstract]. Annual Meeting of Infectious Diseases Society of America, San Francisco, California, 25–28 October 2001:127.

112.  Gilchrist MJ. A national laboratory network for bioterrorism: evolution from a prototype network of laboratories performing routine surveillance. Mil Med. 2000;165:28-31. [PMID: 10920634].[Medline]

113.  National Respiratory and Enteric Virus Surveillance System. Centers for Disease Control and Prevention. August 20, 2001. Accessed at http://www.cdc.gov/ncidod/dvrd/revb/nrevss/index.htm on 29 March 2004.

114.  Tuberculosis Genotyping and Surveillance Network. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/dastlr/tb/tb_tgsn.htm on 24 September 2001.

115.  de Neeling AJ, van Leeuwen WJ, Schouls LM, Schot CS, van Veen-Rutgers A, Beunders AJ, et al. Resistance of staphylococci in The Netherlands: surveillance by an electronic network during 1989-1995. J Antimicrob Chemother. 1998;41:93-101. [PMID: 9511042].[Abstract/Free Full Text]

116.  Martin SM, Bean NH. Data management issues for emerging diseases and new tools for managing surveillance and laboratory data. Emerg Infect Dis. 1995;1:124-8. [PMID: 8903181].[Medline]

117.  Hutwagner LC, Maloney EK, Bean NH, Slutsker L, Martin SM. Using laboratory-based surveillance data for prevention: an algorithm for detecting Salmonella outbreaks. Emerg Infect Dis. 1997;3:395-400. [PMID: 9284390].[Medline]

118.  Grant AD, Eke B. Application of information technology to the laboratory reporting of communicable disease in England and Wales. Commun Dis Rep CDR Rev. 1993;3:R75-8. [PMID: 7693156].[Medline]

119.  Stern L, Lightfoot D, Forsyth JRL. Automated detection of Salmonella outbreaks. In: Witten M, ed. Proceedings of the First World Congress on Computational Medicine, Public Health, and Biotechnology. Singapore: World Scientific; 1996:1395-404.

120.  Stern L, Lightfoot D. Automated outbreak detection: a quantitative retrospective analysis. Epidemiol Infect. 1999;122:103-10. [PMID: 10098792].[Medline]

121.  International surveillance network for the enteric infections: Salmonella and VTEC O157. Enter-net. Accessed at http://www.phls.co.uk/International/Enter-Net/enter-net.htm on 29 November 2001. Now available at http://www.hpa.org.uk/hpa/inter/enter-net_menu.htm.

122.  Crook P, Fisher I. Developments in E.U.-wide surveillance of Salmonella and V.T.E.C. Eurosurveillance Weekly. 2000(48).

123.  Global Outbreak Alert and Response: Report of a WHO Meeting. Geneva, Switzerland: World Health Organization; 26–28 April 2000.

124.  Global Salm-Surv. Accessed at http://www.who.int/emc/diseases/zoo/SALM-SURV/frameset.html on 29 November 2001.

125.  Project ICARE: background and problem description, including earlier ICARE studies. Centers for Disease Control and Prevention, National Nosocomial Infections Surveillance System (NNIS), and Rollins School of Public Health. Accessed at http://www.sph.emory.edu/icare/phase3.html on 21 September 2001. Now available at http://www.sph.emory.edu/ICARE/index.htm.

126.  Surveillance: INSPEAR—International Network for the Study and Prevention of Emerging Antimicrobial Resistance. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/hip/surveill/inspear.htm on 24 September 2001.

127.  Richet HM, Mohammed J, McDonald LC, Jarvis WR. Building communication networks: international network for the study and prevention of emerging antimicrobial resistance. Emerg Infect Dis. 2001;7:319-22. [PMID: 11294732].[Medline]

128.  Davis SR. The state of antibiotic resistance surveillance: an overview of existing activities and new strategies. Mil Med. 2000;165:35-9. [PMID: 10920636].[Medline]

129.  National Antimicrobial Resistance Monitoring System (NARMS): Enteric Bacteria. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/narms/ on 29 March 2004.

130.  Ramphal R, Hoban DJ, Pfaller MA, Jones RN. Comparison of the activity of two broad-spectrum cephalosporins tested against 2,299 strains of Pseudomonas aeruginosa isolated at 38 North American medical centers participating in the SENTRY Antimicrobial Surveillance Program, 1997-1998. Diagn Microbiol Infect Dis. 2000;36:125-9. [PMID: 10705055].[Medline]

131.  Hoban DJ, Doern GV, Fluit AC, Roussel-Delvallez M, Jones RN. Worldwide prevalence of antimicrobial resistance in Streptococcus pneumoniae, Haemophilus influenzae, and Moraxella catarrhalis in the SENTRY Antimicrobial Surveillance Program, 1997-1999. Clin Infect Dis. 2001;32(Suppl 2):S81-93. [PMID: 11320449].

132.  Surveillance for Emerging Antimicrobial Resistance Connected to Healthcare (SEARCH). Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/hip/aresist/search.htm on 9 November 2001.

133.  Sahm DF, Critchley IA, Kelly LJ, Karlowsky JA, Mayfield DC, Thornsberry C, et al. Evaluation of current activities of fluoroquinolones against gram-negative bacilli using centralized in vitro testing and electronic surveillance. Antimicrob Agents Chemother. 2001;45:267-74. [PMID: 11120976].[Abstract/Free Full Text]

134.  Sahm DF, Marsilio MK, Piazza G. Antimicrobial resistance in key bloodstream bacterial isolates: electronic surveillance with the Surveillance Network Database—USA. Clin Infect Dis. 1999;29:259-63. [PMID: 10476722].[Medline]

135.  Vatopoulos AC, Kalapothaki V, Legakis NJ. An electronic network for the surveillance of antimicrobial resistance in bacterial nosocomial isolates in Greece. The Greek Network for the Surveillance of Antimicrobial Resistance. Bull World Health Organ. 1999;77:595-601. [PMID: 10444883].[Medline]

136.  O'Brien TF, Stelling JM. WHONET: removing obstacles to the full use of information about antimicrobial resistance. Diagn Microbiol Infect Dis. 1996;25:162-8. [PMID: 8937840].[Medline]

137.  Stelling JM, O'Brien TF. Surveillance of antimicrobial resistance: the WHONET program. Clin Infect Dis. 1997;24(Suppl 1):S157-68. [PMID: 8994799].

138.  O'Brien TF, Stelling JM. WHONET: an information system for monitoring antimicrobial resistance. Emerg Infect Dis. 1995;1:66 [PMID: 8903165].[Medline]

139.  Riviere JE, Craigmill AL, Sundlof SF. Food animal residue avoidance databank (FARAD): an automated pharmacologic data-base for drug and chemical residual avoidance. J Food Prot. 1986;49:826-30.

140.  Sischo WM, Norman HS, Kiernan NE. Use of the Food Animal Residue Avoidance Databank (FARAD). J Am Vet Med Assoc. 1999;214:344-50. [PMID: 10023394].[Medline]

141.  Foodborne Diseases Active Surveillance Network, 1996. MMWR Morb Mortal Wkly Rep. 1997;46:258-61. [PMID: 9087688].[Medline]

142.  Foodborne Diseases Active Surveillance Network (FoodNet). Emerg Infect Dis. 1997;3:581-3. [PMID: 9368789].[Medline]

143.  Centers for Disease Control and Prevention. FoodNet Surveillance Report for 1999. Final Report. Atlanta, GA: Foodborne Diseases Active Surveillance Network, Centers for Disease Control and Prevention; November 2000.

144.  Preliminary FoodNet data on the incidence of foodborne illnesses—selected sites, United States, 2000. MMWR Morb Mortal Wkly Rep. 2001;50:241-6. [PMID: 11310569].

145.  Wallace DJ, Van Gilder T, Shallow S, Fiorentino T, Segler SD, Smith KE, et al. Incidence of foodborne illnesses reported by the foodborne diseases active surveillance network (FoodNet)-1997. FoodNet Working Group. J Food Prot. 2000;63:807-9. [PMID: 10852576].[Medline]

146.  Stephenson J. New approaches for detecting and curtailing foodborne microbial infections. JAMA. 1997;277:1339-40. [PMID: 9134923].[Abstract/Free Full Text]

147.  Karras DJ. Incidence of foodborne illnesses: preliminary data from the foodborne diseases active surveillance network (FoodNet). Ann Emerg Med. 2000;35:92-3. [PMID: 10613949].[Medline]

148.  Kello D. Epidemiological aspects in food safety. Food Addit Contam. 1990;7(Suppl 1):S5-11. [PMID: 2262040].

149.  Swaminathan B, Barrett TJ, Hunter SB, Tauxe RV. PulseNet: the molecular subtyping network for foodborne bacterial disease surveillance, United States. Emerg Infect Dis. 2001;7:382-9. [PMID: 11384513].[Medline]

150.  Talaska T. A salmonella data bank for routine surveillance and research. Bull World Health Organ. 1994;72:69-72. [PMID: 8131252].[Medline]

151.  Powell SC, Attwell RW. An evaluation of the collection and analysis of epidemiological data for support of food safety control systems. J Food Prot. 1998;61:1170-4. [PMID: 9766070].[Medline]

152.  van Gerwen SJ, de Wit JC, Notermans S, Zwietering MH. An identification procedure for foodborne microbial hazards. Int J Food Microbiol. 1997;38:1-15. [PMID: 9498132].[Medline]

153.  van Gerwen SJ, te Giffel MC, van't Riet K, Beumer RR, Zwietering MH. Stepwise quantitative risk assessment as a tool for characterization of microbiological food safety. J Appl Microbiol. 2000;88:938-51. [PMID: 10849169].[Medline]

154.  McClure PJ, Blackburn CW, Cole MB, Curtis PS, Jones JE, Legan JD, et al. Modelling the growth, survival and death of microorganisms in foods: the UK food micromodel approach. Int J Food Microbiol. 1994;23:265-75. [PMID: 7873330].[Medline]

155.  Garner M, Nunn M. The Australian national animal health information system. Epidemiologie et Sante Animale: Bulletin de l'Association pour l'Etude des Maladies Animales. 1995;27:143-60.

156.  California expands mosquito surveillance [press release]. Sacramento, CA: California Department of Health Services; 15 August 2000. Accessed at http://www.applications.dhs.ca.gov/pressreleases/store/pressreleases/41-00.html on 2 February 2002.

157.  Sanson R. EpiMan—A decision support system for managing a foot and mouth disease epidemic. Surveillance. 1994;21:22-4.

158.  Sanson RL, Morris RS, Stern MW. EpiMAN-FMD: a decision support system for managing epidemics of vesicular disease. Rev Sci Tech. 1999;18:593-605. [PMID: 10588003].[Medline]

159.  Stark KD, Morris RS, Benard HJ, Stern MW. EpiMAN-SF: a decision-support system for managing swine fever epidemics. Rev Sci Tech. 1998;17:682-90. [PMID: 9850539].[Medline]

160.  Blajan L, Chillaud T. The OIE world animal disease information system. Rev Sci Tech. 1991;10:51-87. [PMID: 1760577].[Medline]

161.  Bruckner GK. Monitoring and surveillance systems for animal diseases, taking as models the following diseases: myobacterial infections in animals, Newcastle disease, foot and mouth disease and rabies. Office International Des Epizooties, Comprehensive Reports on Technical Items Presented to the International Committee or to Regional Commissions; 1995. Accessed at http://www.oie.int/eng/publicat/en_themesT.htm on 10 November 2001. Now available at http://www.medvet.umontreal.ca/biblio/gopher/vetjr/otre/1995.html.

162.  Bush E, Gardner I. Animal health surveillance in the United States via the national animal health monitoring system (NAHMS). Epidemiologie et Sante Animale: Bulletin de l'Association pour l'Etude des Maladies Animales. 1995;27:113-26.

163.  Hueston WD. The National Animal Health Monitoring System: addressing animal health information needs in the USA. Prev Vet Med. 1990;8:97-102.

164.  Frank GR, Salman MD, MacVean DW. Use of a disease reporting system in a large beef feedlot. J Am Vet Med Assoc. 1988;192:1063-7. [PMID: 3372332].[Medline]

165.  Losinger WC, Bush EJ, Hill GW, Smith MA, Garber LP, Rodriguez JM, et al. Design and implementation of the United States National Animal Health Monitoring System 1995 National Swine Study. Prev Vet Med. 1998;34:147-59. [PMID: 9604264].[Medline]

166.  West Nile Virus: Statistics, Surveillance, and Control. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/dvbid/westnile/surv&control.htm on 10 September 2001.

167.  Rasley D, Wenzel RP, Massanari RM, Streed S, Hierholzer WJ Jr. Organization and operation of the hospital-infection-control program of the University of Iowa Hospitals and Clinics. Infection. 1988;16:373-8. [PMID: 3220585].[Medline]

168.  Brossette SE, Sprague AP, Hardin JM, Waites KB, Jones WT, Moser SA. Association rules and data mining in hospital infection control and public health surveillance. J Am Med Inform Assoc. 1998;5:373-81. [PMID: 9670134].[Abstract/Free Full Text]

169.  Brossette SE, Sprague AP, Jones WT, Moser SA. A data mining system for infection control surveillance. Methods Inf Med. 2000;39:303-10. [PMID: 11191698].[Medline]

170.  Schifman RB, Palmer RA. Surveillance of nosocomial infections by computer analysis of positive culture rates. J Clin Microbiol. 1985;21:493-5. [PMID: 3988895].[Abstract/Free Full Text]

171.  Smyth ET, McIlvenny G, Barr JG, Dickson LM, Thompson IM. Automated entry of hospital infection surveillance data. Infect Control Hosp Epidemiol. 1997;18:486-91. [PMID: 9247831].[Medline]

172.  Smyth ET, Emmerson AM. Survey of infection in hospitals: use of an automated data entry system. J Hosp Infect. 1996;34:87-97. [PMID: 8910750].[Medline]

173.  About NNIS. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/ncidod/hip/NNIS/@nnis.htm on 24 September 2001.

174.  Emori TG, Culver DH, Horan TC, Jarvis WR, White JW, Olson DR, et al. National nosocomial infections surveillance system (NNIS): description of surveillance methods. Am J Infect Control. 1991;19:19-35. [PMID: 1850582].[Medline]

175.  Evans RS, Burke JP, Classen DC, Gardner RM, Menlove RL, Goodrich KM, et al. Computerized identification of patients at high risk for hospital-acquired infection. Am J Infect Control. 1992;20:4-10. [PMID: 1554148].[Medline]

176.  Evans RS, Larsen RA, Burke JP, Gardner RM, Meier FA, Jacobson JA, et al. Computer surveillance of hospital-acquired infections and antibiotic use. JAMA. 1986;256:1007-11. [PMID: 3735626].[Abstract/Free Full Text]

177.  Joch J, Burkle T, Dudeck J. Decision support for infectious diseases—a working prototype. Stud Health Technol Inform. 2000;77:812-6. [PMID: 11187666].[Medline]

178.  McLaws ML, Caelli M. Pilot testing standardized surveillance: Hospital Infection Standardised Surveillance (HISS). On behalf of the HISS Reference Group. Am J Infect Control. 2000;28:401-5. [PMID: 11114609].[Medline]

179.  McLaws ML, Murphy C, Whitby M. Standardising surveillance of nosocomial infections: the HISS program. Hospital Infection Standardised Surveillance. J Qual Clin Pract. 2000;20:6-11. [PMID: 10821448].[Medline]

180.  Bouam S, Girou E, Brun-Buisson C, Lepage E. Development of a Web-based clinical information system for surveillance of multiresistant organisms and nosocomial infections. Proc AMIA Symp. 1999;:696-700. [PMID: 10566449].

181.  Smyth ET, Barr JG, Bamford KB. Surveillance and management of infection in a haematology unit: use of an in-house clinical database. J Hosp Infect. 1993;25:137-44. [PMID: 7903087].[Medline]

182.  Kahn MG, Bailey TC, Steib SA, Fraser VJ, Dunagan WC. Statistical process control methods for expert system performance monitoring. J Am Med Inform Assoc. 1996;3:258-69. [PMID: 8816348].[Abstract/Free Full Text]

183.  Kahn MG, Steib SA, Dunagan WC, Fraser VJ. Monitoring expert system performance using continuous user feedback. J Am Med Inform Assoc. 1996;3:216-23. [PMID: 8723612].[Abstract/Free Full Text]

184.  Kahn MG, Steib SA, Fraser VJ, Dunagan WC. An expert system for culture-based infection control surveillance. Proc Annu Symp Comput Appl Med Care. 1993;3:171-5. [PMID: 8130456].

185.  Kahn MG, Steib SA, Spitznagel EL, Claiborne DW, Fraser VJ. Improvement in user performance following development and routine use of an expert system. Medinfo. 1995;8:1064-7. [PMID: 8591368].

186.  Burken MI, Zaman AF, Smith FJ. Semi-automated infection control surveillance in a Veterans' Administration Medical Center [Letter]. Infect Control Hosp Epidemiol. 1990;11:410-2. [PMID: 2212582].[Medline]

187.  Madsen KM, Schonheyder HC, Kristensen B, Nielsen GL, Sorensen HT. Can hospital discharge diagnosis be used for surveillance of bacteremia? A data quality study of a Danish hospital discharge registry. Infect Control Hosp Epidemiol. 1998;19:175-80. [PMID: 9552185].[Medline]

188.  Heininger A, Niemetz AH, Keim M, Fretschner R, Doring G, Unertl K. Implementation of an interactive computer-assisted infection monitoring program at the bedside. Infect Control Hosp Epidemiol. 1999;20:444-7. [PMID: 10395153].[Medline]

189.  Samore M, Lichtenberg D, Saubermann L, Kawachi C, Carmeli Y. A clinical data repository enhances hospital infection control. Proc AMIA Annu Fall Symp. 1997;21:56-60. [PMID: 9357588].

190.  Rundio A Jr. Understanding microbiological concepts and computerized surveillance: enhancing professional practice. Health Care Superv. 1994;12:20-7. [PMID: 10132239].[Medline]

191.  Dialysis Surveillance Network (DSN). Atlanta, GA: Centers for Disease Control and Prevention; 1999. Accessed at http://www.cdc.gov/ncidod/hip/DIALYSIS/dsn.htm on 21 September 2001.

192.  Hansen G, Pezzino G. Lessons learned: Kansas system (HAWK). In: Pezzino G, ed. A guide to the implementation of the National Electronic Disease Surveillance System (NEDSS) in state public health agencies. Atlanta, GA: Council of State and Territorial Epidemiologists; 2001:13. Accessed at http://www.cste.org/NNDSSSurvey/Downloads/NEDSS_Book_Final.pdf on 1 April 2004.

193.  Reportable infectious diseases in Kansas: 1999 summary. Topeka, KS: Kansas Department of Health and Environment; 2000. Accessed at http://www.kdhe.state.ks.us/epi/download/disease_summary/dissum99.pdf on 1 April 2004.

194.  Hung J, Posey J, Freedman R, Thorton T. Electronic surveillance of disease states: a preliminary study in electronic detection of respiratory diseases in a primary care setting. Proc AMIA Symp. 1998;:688-92. [PMID: 9929307].

195.  Bernard KW, Graitcer PL, van der Vlugt T, Moran JS, Pulley KM. Epidemiological surveillance in Peace Corps Volunteers: a model for monitoring health in temporary residents of developing countries. Int J Epidemiol. 1989;18:220-6. [PMID: 2722368].[Abstract/Free Full Text]

196.  Unexplained deaths & critical illnesses. Atlanta, GA: Centers for Disease Control and Prevention; 2001. Accessed at http://www.cdc.gov/ncidod/dbmd/diseaseinfo/unexplaineddeaths_t.htm on 24 September 2001.

197.  Yokoe DS, Subramanyan GS, Nardell E, Sharnprapai S, McCray E, Platt R. Supplementing tuberculosis surveillance with automated data from health maintenance organizations. Emerg Infect Dis. 1999;5:779-87. [PMID: 10603211].[Medline]

198.  Maggini M, Salmaso S, Alegiani SS, Caffari B, Raschetti R. Epidemiological use of drug prescriptions as markers of disease frequency: an Italian experience. J Clin Epidemiol. 1991;44:1299-307. [PMID: 1753261].[Medline]

199.  TheDataWeb. Centers for Disease Control and Prevention. Accessed at http://www.cdc.gov/programs/research20.htm on 24 September 2001. Now available at http://www.thedataweb.org/index.html.

200.  Maqbool S. Pakistan is high-risk country for epidemics. The News. 11 July 2001. Accessed at http://www.jang.com.pk/thenews/jul2001-daily/11-07-2001/metro/i3.htm on 4 November 2001. Now available at http://lists.isb.sdnpk.org/permail/health-list/2001-July/000723.html.

201.  The work of WHO in the Eastern Mediterranean region. Section 5.2: Control of other communicable diseases. In: World Health Organization. Annual Report of the Regional Director. Geneva: World Health Organization; 2000. Accessed at http://www.emro.who.int/Rd/AnnualReports/2000/chapter5-2.htm on 4 November 2001.

202.  Surveillance: National Surveillance System for Health Care Workers. Atlanta, GA: Centers for Disease Control and Prevention; 2000. Accessed at http://www.cdc.gov/ncidod/hip/SURVEILL/nash.htm on 24 September 2001.

203.  Osaka K, Inouye S, Okabe N, Taniguchi K, Izumiya H, Watanabe H, et al. Electronic network for monitoring travellers' diarrhoea and detection of an outbreak caused by Salmonella enteritidis among overseas travellers. Epidemiol Infect. 1999;123:431-6. [PMID: 10694153].[Medline]

204.  Kaufmann AF, Meltzer MI, Schmid GP. The economic impact of a bioterrorist attack: are prevention and postattack intervention programs justifiable? Emerg Infect Dis. 1997;3:83-94. [PMID: 9204289].[Medline]

205.  Wein LM, Craft DL, Kaplan EH. Emergency response to an anthrax attack. Proc Natl Acad Sci U S A. 2003;100:4346-51. [PMID: 12651951].[Abstract/Free Full Text]

206.  Ma JZ, Ackerman E. Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. II. Durations and peaks. Int J Biomed Comput. 1993;32:255-68. [PMID: 8514440].[Medline]

207.  Ma JZ, Ackerman E, Yang JJ. Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. I. Illness attack rates. Int J Biomed Comput. 1993;32:237-53. [PMID: 8514439].[Medline]

208.  Ma JZ, Peterson DR, Ackerman E. Parameter sensitivity of a model of viral epidemics simulated with Monte Carlo techniques. IV. Parametric ranges and optimization. Int J Biomed Comput. 1993;33:297-311. [PMID: 8307660].[Medline]

209.  Gonzalez-Guzman J. An epidemiological model for direct and indirect transmission of typhoid fever. Math Biosci. 1989;96:33-46. [PMID: 2520190].[Medline]