How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis

View larger version (34K): [in this window] [in a new window]   Figure 1. Overall process for determining updating status. Includes the search protocols to identify candidate new trials, application of criteria from the original review to identify eligible new trials, meta-analytic pooling of new results with previous meta-analytic results, and identification of new systematic reviews on the same topic or "pivotal trials" (published in 1 of the 5 highest-impact general medical journals or more than 3 times the sample size of the previous largest trial) that met any of our criteria for qualitative signals for updating. An individual reviewer reached a tentative conclusion about the presence of quantitative and qualitative signals for updating, but each review was discussed in detail by the project team to reach a final consensus decision. For reviews that did not have any signals for updating, the group also decided whether the searches had been adequate or whether more comprehensive searching for new trials might be required, including more detailed electronic searching and hand-searching of ACP Journal Club for new trials relevant to the original review.

View larger version (50K): [in this window] [in a new window]   Appendix Figure 1. Screening of potential systematic reviews for inclusion in cohort. R = reason; RCT = randomized, controlled trial.

View larger version (12K): [in this window] [in a new window]   Figure 2. Overall survival time (95% CI) free of signals for updating. The immediate decrease in survival at time zero reflects the 7 systematic reviews for which signals for updating had already occurred at the time of publication. The low number of reviews at risk after 10 years reflects the fact that the sample spanned 1995 to 2005 and censoring occurred on 1 September 2006. Thus, only reviews published before September 1996 and having no signals for updating could have more than 10 years of observation.

View larger version (13K): [in this window] [in a new window]   Appendix Figure 2. Survival of the original systematic review by clinical topic area. Stratified by cardiovascular reviews (n = 20) versus reviews on all other topics (n = 80).

View larger version (12K): [in this window] [in a new window]   Appendix Figure 3. Kaplan–Meier plot showing the effect on survival of increasing the total number of patients by more than a factor of 2. Ratio of new total sample size to old total sample size is >2, which occurred for 25% of systematic reviews in the cohort.