Where Do Questions Come From?

Questions arise constantly in routine clinical practice, provided that clinicians are prepared to admit their own level of uncertainty or lack of knowledge [1]. The most relevant questions are often asked directly or indirectly by patients [2, 3]. Most clinical encounters generate questions about diagnosis ("What do I have, doctor?"), etiology ("Is it because I did X?"), prognosis ("How long do I have?"), or treatment or prevention ("Will Y do me any good?"). The ways in which different clinicians manage the same clinical problem vary widely, both within and between countries; these variations should raise questions about which management policies are best. The introduction of new treatments or diagnostic tests should always lead to the question, "Are they better than what we have already?" Researchers who are planning new studies must consider whether the answer to their question already exists, whereas purchasers of health care must ask which health care packages they should buy. Systematic reviews of all available information would help in each of these situations.

Choosing an Important Question

The number of possible questions for systematic reviews is limitless, but the time and resources with which to answer them are limited. Therefore, researchers who undertake systematic reviews must choose the most important questions. This is difficult because the importance of a question varies according to the perspective of the person asking it. For example, individual patients will probably regard questions about their conditions as the most important, regardless of how common or severe that condition is, whereas cardiologists will prefer questions about ischemic heart disease to those about migraine. Cancer and vascular disease are particularly important to persons in developed countries, whereas infectious diseases are more important to persons in developing countries. These differences highlight the need for each specialty to organize its own systematic reviews. Several factors should be considered when setting priorities for doing systematic reviews [4-8] (Table 1).

Important questions may deal with conditions that have a major effect on patients or on persons who care for them, such as conditions associated with serious illness (diabetes), death (cancer), or with impaired quality of life (back pain). Very common conditions may have a major effect on society even if they are minor and short-lived (such as the common cold, which results in many aggregate days of missed work). Most health care interventions have potentially harmful effects and should be carefully evaluated. It is particularly important to assess expensive interventions (for example, interferon therapy for multiple sclerosis [9]); widely used interventions, because these may cause widespread harm (for example, lignocaine in acute myocardial infarction [10]); and simple interventions that could be widely used if they are shown to be effective (such as steroids for preterm delivery [11]).

The most useful reviews are those that can improve clinical practice. Widespread change is more likely to occur if collective uncertainty exists; this uncertainty is often reflected in variations in practice. Asking a question that has already been answered by common sense or by powerful empirical evidence is of little use unless evidence suggests that the existing answer is wrong. It is also difficult to influence well-established practices, even if the evidence for their utility is poor (for example, cervical screening programs); new technologies must therefore be assessed early in their development. Some researchers consider it more important to concentrate on questions for which data are known to exist [4, 5], but this reasoning may be flawed. A thorough search often identifies previously unknown data or a lack of useful data for some important questions, thereby demonstrating the need for future research.

Although the factors in Table 1 primarily concern the assessment of health technology, most also apply to prioritizing questions about risk or prognostic factors (that is, the frequency, reversibility, and measurability of the factors). Clearly, many of the factors are relatively subjective (such as effect on the patient), thereby underscoring the need to involve patients in setting priorities [2, 3].

Formulating the Question

Having decided that a question is worth asking, the next step is to formulate it adequately (Figure 1). Clinical questions should have four basic components [12]: the type of person involved, the type of exposure that the person experiences (be it a risk factor, prognostic factor, intervention, or diagnostic test), the type of control with which the exposure is being compared, and the outcomes to be addressed.

View larger version (14K): [in this window] [in a new window]   Figure 1. Examples of poorly formulated and well-formulated questions.

A clearly formulated question helps define the criteria that studies must meet to be included in the review (Figure 2). These inclusion criteria can be divided into five categories, shown in Table 2. Each component must be carefully defined to strike a balance between making the definition too specific to be workable and making it too broad to be useful. In Figure 2, the definition of ischemic stroke could be "a stroke with an occluded artery on angiography"; this definition, however, would exclude most studies because few patients with stroke undergo angiography. Alternatively, it could be defined simply as "ischemic cerebrovascular disease," but it then becomes unclear whether patients with transient ischemic attacks and vascular dementia are included. Some complex exposures or interventions can be difficult to define precisely, especially if they are nonpharmacologic (for example, how does one define speech therapy?). In this situation, it may be useful to begin with a broad working definition that can later be refined and to test the reproducibility of the definition among different types of persons (for example, speech therapy could be defined as "therapy given by a qualified speech therapist to improve language function"). It is also important to remember that definitions may very among countries and among studies (for example, the term chronic cerebrovascular disease is sometimes used to mean "vascular dementia").

View larger version (25K): [in this window] [in a new window]   Figure 2. How a well-formulated question guides the review process. CT = computed tomography; RCT = randomized, controlled trial. * = see Appendix Table.

View this table: [in this window] [in a new window]   Appendix Table. Example of a MEDLINE Search for Studies of Anticoagulant Agents in Acute Ischemic Stroke

The outcomes to be assessed should be clinically relevant to the patient [13]. They must consider the perspective of the patient because physicians and patients often do not agree on what issues are important [2, 3]. Indirect or surrogate outcome measures, such as laboratory or radiologic results, should be avoided or interpreted with extreme caution because they rarely predict clinically important outcomes accurately [14]. Surrogate measures may tell you how a treatment might work but not whether it actually does work. Many treatments reduce the risk for a surrogate outcome but have no effect or have harmful effects on clinically relevant outcomes, and some treatments have no effect on surrogate measures but improve clinical outcomes [14]. For example, lignocaine has been shown to suppress ventricular arrhythmias after myocardial infarction [15] but increases case-fatality rates [10]. Systematic reviews of treatments should measure adverse effects as well as beneficial effects. Reviewers may also wish to record data on costs to perform an economic evaluation, although this requires expert guidance [16]. In addition to defining the outcomes that are to be measured, the inclusion criteria must state when the outcomes should be measured. For chronic diseases, outcomes that are assessed after a short follow-up period may not reflect long-term outcome.

Reviews should always focus on the best available evidence: that is, studies to be included in a review should use methods that provide the least biased answer to the question asked. Therefore, systematic reviews of treatment and prevention should include randomized, controlled trials [17], particularly those that use a well-concealed method of allocation [18]. Reviews of diagnostic tests should include studies that independently compared one or more tests with an adequate gold standard [19, 20]. Reviews of prognosis should include cohort studies in which a representative sample of patients was entered at a similar point in the course of disease [21]. Finally, reviews of risk factors could include relevant case–control, cohort, or ecologic studies, ideally with multivariate analysis to adjust for other known risk factors. If no studies providing the best level of evidence are found, it may be appropriate to consider other levels of evidence. For example, no evidence from randomized, controlled trials suggests that lying babies supine rather than prone prevents the sudden infant death syndrome, but a substantial body of observational data suggests that this is the case [22]. Other details that must be considered are the requirement for blinding in randomized, controlled trials [18] and studies of diagnostic tests [19, 20] and the presence of confounding factors (Table 2).

How Broad Should the Inclusion Criteria Be?

The scope of the question and, hence, the inclusion criteria can be relatively broad or narrow. The choice of inclusion criteria depends on several factors. Questions must be clinically relevant: A broad question ("Has chemotherapy improved cancer survival?") will not help a clinician manage a patient with a particular tumor because of marked differences in the responses of different tumors. Inclusion criteria must also be clinically sensible. If certain features of the patients or exposures are believed to significantly affect outcome, these features must be taken into account. For example, the effects of anti-coagulation therapy will probably differ in patients with hemorrhagic stroke compared with patients with ischemic stroke; thus, it is sensible to restrict the question in Figure 2 to ischemic strokes. However, narrow inclusion criteria limit the amount of data in the review and thereby increase the risk for false-positive and false-negative results [23, 24]. A narrow question can be regarded as a subgroup of a broader question and can lead to the same problems generally found in subgroup analysis [25, 26]. Narrow inclusion criteria also preclude studying appropriate and clinically important subgroups in the context of a larger data set. A review of the effects of 5000 units of unfractionated heparin given twice daily in acute stroke will not allow comparisons of the effects of different anticoagulant agents or different doses of heparin. Broad inclusion criteria increase the risk for finding heterogeneity (that is, significant variation in the results of different studies), thereby making analysis and interpretation of the results more difficult [27]. If no heterogeneity is found even when broad inclusion criteria are used, the results are more generalizable. Heterogeneity among studies can be useful, however, because it allows the researcher to study what caused it and generate new hypotheses [27, 28]. Broad reviews can summarize large amounts of information in a single article; this may be more useful for readers but may require greater resources.

Although the inclusion criteria must be set before data collection begins, they should be flexible, provided that care is taken to avoid making changes that would be likely to introduce bias. Inclusion criteria should not be changed on the basis of the results of individual trials. It may, however, be reasonable to change the criteria if alternative, acceptable ways of defining the study population or intervention are discovered. Narrow criteria may also need to be broadened or broad criteria may need to be narrowed, depending on the amount of data found.

Locating Relevant Studies

As is the case with primary research studies, flaws in data collection can invalidate the results of a systematic review. As many relevant primary studies as possible (as resources allow) must be collected to minimize random error and bias. The first step is to decide which types of article must be retrieved with regard to the language of publication and publication status. For practical reasons, many systematic reviews restrict articles to the English language, but this practice is hard to justify. Existing evidence does not suggest that the quality of research varies by language of publication [29], and language restrictions can alter the results of systematic reviews by excluding available data [30]. Some researchers [31] think that unpublished studies should be excluded because they have not been peer reviewed and therefore may have unreliable results. However, the most important influence on publication status is probably not scientific rigor but the nature of the results themselves. Publication bias- the selective publication of studies based on the direction and strength of their results [32] -affects all types of studies (including randomized, controlled trials [32-34]), but it seems to be a greater problem for small, nonrandomized studies [32, 33, 35]. Because fewer studies with negative or null results are published than studies with larger, more positive results [32-34], reviews that exclude unpublished work are likely to overestimate the relation between the exposure and the outcome. As a consequence, treatment effects may be overestimated, making ineffective treatments seem effective [36, 37]. Most researchers who do systematic reviews therefore think that unpublished studies should be included; if necessary, the results can be reanalyzed without the unpublished data [38].

Even among published studies, the nature of results varies according to the type of publication. Many primary studies are published as abstracts in conference proceedings, but only 50% of these go on to be published in full [39]. Many studies that are published only as abstracts do not have statistically significant results; thus, excluding abstracts from systematic reviews may again limit the amount of data included in the review and introduce bias [39]. Similar findings may apply to studies that were only published as letters or dissertations [40]. Further data must be sought from the authors of letters and abstracts to determine whether they are eligible for inclusion in the review.

Search Strategies

Several complementary strategies can identify studies that are relevant to a systematic review (Table 3). Whichever methods are used, they must be reported in sufficient detail to allow replication. Again, a well-formulated question helps define the best search strategy (Figure 2).

The most commonly used strategy is searching electronic databases, such as MEDLINE or EMBASE. This method allows relatively quick access to large amounts of literature. Superficially, electronic searching seems simple: Enter a few appropriate terms, and the database should give back all the appropriate studies. In practice, however, electronic database searching is much more difficult. Studies have shown that depending on the topic, a MEDLINE search will identify only 32% to 91% of randomized, controlled trials published in journals that are indexed by MEDLINE [41]. This is in part due to inadequate indexing (attempts to improve the indexing of randomized, controlled trials are now under way [41, 42]) and in part to use of incorrect terms in the search strategy [41, 43]. Reviewers must therefore take the time to plan their search systematically [43] and get help from persons who are experienced in using particular databases, such as medical librarians.

The first step in developing an electronic search is to include terms that refer to the disease or condition of interest (Appendix Table). Depending on the number of articles retrieved, the reviewer can restrict this search by combining it with searches for the exposure and study design of interest, then further restrict it to studies in humans. Because of the unreliability of indexing, the final search must include both controlled vocabulary terms, which vary with each database, and free text terms [41, 44]. The best terms to use in the search can be determined by studying previously identified articles that meet the inclusion criteria. A provisional search can then be run on the most recent years and modified according to the articles found. Ideally, the performance of the electronic search should be validated by comparing its sensitivity against results obtained by doing a manual search of selected journals. For reviews that include randomized, controlled trials, an optimal MEDLINE search for randomized, controlled trials has been developed on behalf of the Cochrane Collaboration; a similar search should soon be available for EMBASE [45]. As the number of terms in an electronic search increases, the number of both relevant and irrelevant articles identified increases. The reviewers must find the optimal balance between the two given available resources.

Even with an optimal search strategy, an electronic search will not identify articles in journals that are not indexed in the database (MEDLINE, for example, indexes about 4000 of 16000 biomedical journals [41] and does not index issues before 1966); articles published in conference proceedings, because these are rarely indexed even if the proceedings are published in journals; unpublished articles; or articles published in books or dissertations. To minimize these problems, different databases can be searched to increase the coverage of journals; EMBASE, for example, indexes more than 1000 journals that are not found in MEDLINE. Many other general and subject-specific databases also exist, such as BIOSIS, CINAHL, PsychLit, and CancerLit. Other databases cover publications in languages other than English [46], conference proceedings (The International Scientific and Technical Proceedings database), dissertations (Index to UK Theses, Dissertation Abstracts), and unpublished literature (SIGLE [System for Information on the Grey Literature in Europe]) [41]. However, these databases may be unavailable or costly to access, and each requires the development of a specific search strategy.

Given the problems of electronic searching, reviewers should use additional methods. A manual, page-by-page search of important journals and conference proceedings overcomes the problems of lack of coverage and poor indexing in databases. However, manual searching is very time- and labor-intensive, particularly if many journals must be searched (although volunteers can be trained to do this [47]). The Cochrane Collaboration is coordinating the manual searching of journals for randomized, controlled trials to minimize duplication of effort [45] and has also produced guidelines for quality control [48]. Reference lists of studies and existing reviews are also useful sources of studies, but they do not identify a representative sample. Reference (citation) bias results in more favorable studies being cited more frequently [49]. Reference tracking systems, such as the Science Citation Index, can also be used to identify the articles that quote important references. Many registries of studies already exist [45, 50] and should be searched. Current-awareness publications, such as Current Contents, are also useful to search because they include recently published research that has not yet been included in electronic databases.

Identifying unpublished studies is very difficult. Contacting companies can help identify industry-supported studies that involved particular drugs or appliances, but some companies may not be willing to share their data. Surveys of authors of previous research, experts in the field, and colleagues may also identify unpublished studies. However, these surveys are time-consuming and expensive and the retrieval rates can be very low. A survey of 42 000 obstetricians and pediatricians identified only 18 unpublished randomized, controlled trials in prenatal and obstetric care that had been completed more than 2 years previously [51]. One solution to the problems of identifying unpublished studies would be to require registration of all research projects at their inception. Indeed, calls for such registration of clinical trials have repeatedly been made [32, 33, 41, 51-53], but major logistic and political hurdles must be overcome before this can become a reality [51, 53]. In the meantime, researchers should be encouraged to submit their results for publication and to comply with requests for unpublished material because it is unethical not to do so [54].

The use of all the sources shown in Table 3 would create a comprehensive list of studies but would be costly in terms of time and money. Reviewers with limited resources must select the methods with the highest yield [55]. Collaboration and avoidance of duplication are essential for decreasing the workload. These facts are recognized by such groups as the Cochrane Collaboration, which is coordinating an international effort to develop subject-specific registries of controlled clinical trials [45, 56]. These registries are being combined [57] and, along with efforts to improve indexing of trials in MEDLINE [41, 42], may lead to the creation of a much-needed registry of all controlled clinical trials.

Conclusion

Top Conclusion Author & Article Info References

Considerable care is usually taken in the design and conduct of primary research studies. The same care should be taken in developing the protocol and the methods for a systematic review. The quality of a review largely depends on the time and attention spent at the outset on choosing an important question, formulating that question carefully, and planning the best ways to identify relevant primary data. For a researcher planning a systematic review, these investments will pay dividends later in terms of avoiding wasted time and effort and increasing the chance that the final review is scientifically, statistically, and clinically meaningful.

Key Points To Remember

Choose an important, well-focused question

Refine the four major components of the question (people, exposure, control group, outcomes)

Set clear, workable inclusion criteria

Take time to plan a sensible and thorough search strategy

Use multiple overlapping sources of data

Ensure that clinical and methodologic expertise and support are available