Systematic Reviews: Time to Address Clinical and Policy Relevance As Well As Methodological Rigor

Compared with other study designs, well-done randomized trials provide the most valid estimate of the benefits of health interventions because they minimize bias. Systematic reviews of randomized trials identify all studies that have addressed a particular question, and meta-analyses combine the results by using methods that minimize bias (1). Many consider systematic reviews to be the best source of information for making clinical and policy decisions.

Such groups as the Cochrane Collaboration have established standards for the conduct and reporting of systematic reviews (2). In this issue, Shojania and colleagues (3) extend this tradition of subjecting research methods to close scrutiny. They describe how quickly the conclusions of 100 systematic reviews published in ACP Journal Club became out of date because of newly completed randomized trials that changed either the statistical significance or the magnitude of the summary treatment effect when they were added to the original review. Fifty percent of reviews were out of date within 5.5 years after publication and 23% were out of date within 2 years. Not surprisingly, the results were most likely to change for cardiovascular interventions (presumably because of the large number of trials in cardiovascular medicine) or if the trials in the original review were heterogeneous.

Unfortunately, Shojania and colleagues' findings do not allow authors to accurately identify the time at which their specific systematic review will be out of date. Because outdated reviews can be misleading, we believe that researchers should update a literature search annually to identify studies that might change the results of the original review, unless a search of clinical trial registries assures them that no randomized trials addressing the topic of the review are ongoing. Because the updating search should use …