 |
 |
|
Home |
Current Issue |
Past Issues |
In the Clinic |
ACP Journal Club |
CME |
Collections |
Audio/Video |
Mobile |
Subscribe |
Tools |
Help |
ACP Online
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
16 March 1999 | Volume 130 Issue 6 | Pages 515-524
Physicians are often asked to make prognostic assessments but often worry that their assessments will prove inaccurate. Prognostic systems were developed to enhance the accuracy of such assessments. This paper describes an approach for evaluating prognostic systems based on the accuracy (calibration and discrimination) and generalizability (reproducibility and transportability) of the system's predictions. Reproducibility is the ability to produce accurate predictions among patients not included in the development of the system but from the same population. Transportability is the ability to produce accurate predictions among patients drawn from a different but plausibly related population. On the basis of the observation that the generalizability of a prognostic system is commonly limited to a single historical period, geographic location, methodologic approach, disease spectrum, or follow-up interval, we describe a working hierarchy of the cumulative generalizability of prognostic systems.
This approach is illustrated in a structured review of the Dukes and Jass staging systems for colon and rectal cancer and applied to a young man with colon cancer. Because it treats the development of the system as a "black box" and evaluates only the performance of the predictions, the approach can be applied to any system that generates predicted probabilities. Although the Dukes and Jass staging systems are discrete, the approach can also be applied to systems that generate continuous predictions and, with some modification, to systems that predict over multiple time periods. Like any scientific hypothesis, the generalizability of a prognostic system is established by being tested and being found accurate across increasingly diverse settings. The more numerous and diverse the settings in which the system is tested and found accurate, the more likely it will generalize to an untested setting.
Author and Article Information
From Pittsburgh Veterans Affairs Health Care System and University of Pittsburgh, Pittsburgh, Pennsylvania; University of California, San Francisco, and San Francisco Veterans Affairs Medical Center, San Francisco, California; and School of Medicine and Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, Pennsylvania.
Grant Support: This work is derived in part from a doctoral dissertation on health systems by Dr. Justice for the Graduate Group in Managerial Science and Applied Economics, University of Pennsylvania (1996), which was supported by Agency for Health Care Policy and Research National Research Service Award #5 T32 HS00009 from the University of Pennsylvania.
Requests for Reprints: Amy C. Justice, MD, PhD, Pittsburgh Veterans Affairs Health Care System, Section of General Internal Medicine (11E), University Drive C, Pittsburgh, PA 15240.
Current Author Addresses: Dr. Justice: Pittsburgh Veterans Affairs Health Care System, Section of General Internal Medicine (11E-120), University Drive C, Pittsburgh, PA 15240.
Dr. Covinsky: Division of Geriatrics, 111G, Veterans Affairs Medical Center, 4150 Clement Street, San Francisco, CA 94121.
Dr. Berlin: Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, 423 Guardian Drive, Room 815, Blockley Hall, Philadelphia, PA 19104-6021. ACADEMIA AND CLINIC
Assessing the Generalizability of Prognostic Information
This article has been cited by other articles:
|
|
 |
|
  K. J. M. Janssen, C. J. Kalkman, D. E. Grobbee, G. J. Bonsel, K. G. M. Moons, and Y. Vergouwe The Risk of Severe Postoperative Pain: Modification and Validation of a Clinical Prediction Rule Anesth. Analg., October 1, 2008; 107(4): 1330 - 1339. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. van der Steen, R. Kruse, K. Szafara, D. Mehr, G van der Wal, M. Ribbe, and R. D'Agostino Benefits and pitfalls of pooling datasets from comparable observational studies: combining US and Dutch nursing home studies Palliative Medicine, September 1, 2008; 22(6): 750 - 759. [Abstract] [PDF]
|
|
|
|
 |
|
 P M Rothwell Prognostic models Practical Neurology, August 1, 2008; 8(4): 242 - 253. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 I. M. Bennett, A. Coco, J. C. Coyne, A. J. Mitchell, J. Nicholson, E. Johnson, M. Horst, and S. Ratcliffe Efficiency of a Two-Item Pre-Screen to Reduce the Burden of Depression Screening in Pregnancy and Postpartum: An IMPLICIT Network Study J Am Board Fam Med, July 1, 2008; 21(4): 317 - 325. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. Sanabria, A. L. Carvalho, J. G. Vartanian, J. Magrin, M. K. Ikeda, and L. P. Kowalski Validation of the Washington University Head and Neck Comorbidity Index in a Cohort of Older Patients Arch Otolaryngol Head Neck Surg, June 1, 2008; 134(6): 603 - 607. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. Verduijn, N. Peek, N. F. de Keizer, E.-J. van Lieshout, A.-C. J.M. de Pont, M. J. Schultz, E. de Jonge, and B. A.J.M. de Mol Individual and Joint Expert Judgments as Reference Standards in Artifact Detection J. Am. Med. Inform. Assoc., March 1, 2008; 15(2): 227 - 234. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. A. Zisman, A. S. Karlamangla, S. M. Kawut, O. A. Shlobin, R. Saggar, D. J. Ross, M. I. Schwarz, J. A. Belperio, A. Ardehali, J. P. Lynch III, et al. Validation of a Method To Screen for Pulmonary Hypertension in Advanced Idiopathic Pulmonary Fibrosis Chest, March 1, 2008; 133(3): 640 - 645. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 MRC CRASH Trial Collaborators Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients BMJ, February 23, 2008; 336(7641): 425 - 429. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. Kotz, P. Nelemans, C. P. van Schayck, and G. J. Wesseling External validation of a COPD diagnostic questionnaire Eur. Respir. J., February 1, 2008; 31(2): 298 - 303. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. May, J. A C Sterne, M. Shipley, E. Brunner, R. d'Agostino, P. Whincup, Y. Ben-Shlomo, A. Carr, B. Ledergerber, J. D Lundgren, et al. A coronary heart disease risk model for predicting the effect of potent antiretroviral therapy in HIV-1 infected men Int. J. Epidemiol., December 1, 2007; 36(6): 1309 - 1318. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. F Beattie and R. M Nelson Evaluating Research Studies That Address Prognosis for Patients Receiving Physical Therapy Care: A Clinical Update Physical Therapy, November 1, 2007; 87(11): 1527 - 1535. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. P. Vandenbroucke, E. v. Elm, D. G. Altman, P. C. Gotzsche, C. D. Mulrow, S. J. Pocock, C. Poole, J. J. Schlesselman, M. Egger, and for the STROBE initiative Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration Ann Intern Med, October 16, 2007; 147(8): W-163 - W-194. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. F. Symmans, F. Peintinger, C. Hatzis, R. Rajan, H. Kuerer, V. Valero, L. Assad, A. Poniecka, B. Hennessy, M. Green, et al. Measurement of Residual Breast Cancer Burden to Predict Survival After Neoadjuvant Chemotherapy J. Clin. Oncol., October 1, 2007; 25(28): 4414 - 4422. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. Teramukai, K. Ochiai, H. Tada, and M. Fukushima PIEPOC: A New Prognostic Index for Advanced Epithelial Ovarian Cancer Japan Multinational Trial Organization OC01-01 J. Clin. Oncol., August 1, 2007; 25(22): 3302 - 3306. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. K. Inouye, Y. Zhang, R. N. Jones, D. K. Kiely, F. Yang, and E. R. Marcantonio Risk Factors for Delirium at Discharge: Development and Validation of a Predictive Model Arch Intern Med, July 9, 2007; 167(13): 1406 - 1413. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. M. Marzec, J. D. Christie, S. P. Reddy, A. E. Jedlicka, H. Vuong, P. N. Lanken, R. Aplenc, T. Yamamoto, M. Yamamoto, H.-Y. Cho, et al. Functional polymorphisms in the transcription factor NRF2 in humans increase the risk of acute lung injury FASEB J, July 1, 2007; 21(9): 2237 - 2246. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. F. Imperiale, J. A. Dominitz, D. T. Provenzale, L. P. Boes, C. M. Rose, J. C. Bowers, B. S. Musick, F. Azzouz, and S. M. Perkins Predicting Poor Outcome From Acute Upper Gastrointestinal Hemorrhage Arch Intern Med, June 25, 2007; 167(12): 1291 - 1296. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. N. Wijeysundera, K. Karkouti, J.-Y. Dupuis, V. Rao, C. T. Chan, J. T. Granton, and W. S. Beattie Derivation and Validation of a Simplified Predictive Index for Renal Replacement Therapy After Cardiac Surgery JAMA, April 25, 2007; 297(16): 1801 - 1809. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M May, D A Lawlor, P Brindle, R Patel, and S Ebrahim Cardiovascular disease risk assessment in older women: can we improve on Framingham? British Women's Heart and Health prospective cohort study Heart, October 1, 2006; 92(10): 1396 - 1401. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 O. Birim, A. P. Kappetein, M. Waleboer, J. P.A. Puvimanasinghe, M. J.C. Eijkemans, E. W. Steyerberg, M. I.M. Versteegh, and A. J.J.C. Bogers Long-term survival after non-small cell lung cancer surgery: Development and validation of a prognostic model with a preoperative and postoperative mode. J. Thorac. Cardiovasc. Surg., September 1, 2006; 132(3): 491 - 498. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. C. Zgibor, G. A. Piatt, K. Ruppert, T. J. Orchard, and M. S. Roberts Deficiencies of Cardiovascular Risk Prediction Models for Type 1 Diabetes Diabetes Care, August 1, 2006; 29(8): 1860 - 1865. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. S. Vasan Biomarkers of Cardiovascular Disease: Molecular Basis and Practical Considerations Circulation, May 16, 2006; 113(19): 2335 - 2362. [Full Text] [PDF]
|
|
|
|
|
|
S. J. Lee, K. Lindquist, M. R. Segal, and K. E. Covinsky Development and Validation of a Prognostic Index for 4-Year Mortality in Older Adults JAMA, February 15, 2006; 295(7): 801 - 808. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. Aujesky, P.-M. Roy, C. P. Le Manach, F. Verschuren, G. Meyer, D. S. Obrosky, R. A. Stone, J. Cornuz, and M. J. Fine Validation of a model to predict adverse outcomes in patients with pulmonary embolism Eur. Heart J., February 2, 2006; 27(4): 476 - 481. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
H. M. Krumholz, R. G. Brindis, J. E. Brush, D. J. Cohen, A. J. Epstein, K. Furie, G. Howard, E. D. Peterson, S. S. Rathore, S. C. Smith Jr, et al. Standards for Statistical Models Used for Public Reporting of Health Outcomes: An American Heart Association Scientific Statement From the Quality of Care and Outcomes Research Interdisciplinary Writing Group: Cosponsored by the Council on Epidemiology and Prevention and the Stroke Council Endorsed by the American College of Cardiology Foundation Circulation, January 24, 2006; 113(3): 456 - 462. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 F. A. Medeiros, R. N. Weinreb, P. A. Sample, C. F. Gomi, C. Bowd, J. G. Crowston, and L. M. Zangwill Validation of a Predictive Model to Estimate the Risk of Conversion From Ocular Hypertension to Glaucoma Arch Ophthalmol, October 1, 2005; 123(10): 1351 - 1360. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. Oudega, A. W. Hoes, and K. G.M. Moons The Wells Rule Does Not Adequately Rule Out Deep Venous Thrombosis in Primary Care Patients Ann Intern Med, July 19, 2005; 143(2): 100 - 107. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. M. Egan and R. M. Kotloff Pro/Con Debate: Lung Allocation Should Be Based on Medical Urgency and Transplant Survival and Not on Waiting Time* Chest, July 1, 2005; 128(1): 407 - 415. [Full Text] [PDF]
|
|
|
|
 |
|
 C. C. Hunault, J. S.E. Laven, I. A.J. van Rooij, M. J.C. Eijkemans, E. R. te Velde, and J. D. F. Habbema Prospective validation of two models predicting pregnancy leading to live birth among untreated subfertile couples Hum. Reprod., June 1, 2005; 20(6): 1636 - 1641. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. Oudega, K. G. Moons, and A. W Hoes Limited value of patient history and physical examination in diagnosing deep vein thrombosis in primary care Fam. Pract., February 1, 2005; 22(1): 86 - 91. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. B. Bach, E. B. Elkin, U. Pastorino, M. W. Kattan, A. I. Mushlin, C. B. Begg, and D. M. Parkin Benchmarking Lung Cancer Mortality Rates in Current and Former Smokers Chest, December 1, 2004; 126(6): 1742 - 1749. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. L. Graham and D. A. Cook Prediction of Risk of Death Using 30-Day Outcome: A Practical End Point for Quality Auditing in Intensive Care Chest, April 1, 2004; 125(4): 1458 - 1466. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M.J. Wildman, J. O'Dea, O. Kostopoulou, M. Tindall, S. Walia, and Z. Khan Variation in intubation decisions for patients with chronic obstructive pulmonary disease in one critical care network QJM, August 1, 2003; 96(8): 583 - 591. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E Giannini, F Botta, P Borro, D Risso, P Romagnoli, A Fasoli, M R Mele, E Testa, C Mansi, V Savarino, et al. Platelet count/spleen diameter ratio: proposal and validation of a non-invasive parameter to predict the presence of oesophageal varices in patients with liver cirrhosis Gut, August 1, 2003; 52(8): 1200 - 1205. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. B. Lamont and N. A. Christakis Complexities in Prognostication in Advanced Cancer: "To Help Them Live Their Lives the Way They Want to" JAMA, July 2, 2003; 290(1): 98 - 104. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. Newcomer, K. E. Covinsky, T. Clay, and K. Yaffe Predicting 12-Month Mortality for Persons With Dementia J. Gerontol. B. Psychol. Sci. Soc. Sci., May 1, 2003; 58(3): S187 - 198. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. B. Bach, M. W. Kattan, M. D. Thornquist, M. G. Kris, R. C. Tate, M. J. Barnett, L. J. Hsieh, and C. B. Begg Variations in Lung Cancer Risk Among Smokers J Natl Cancer Inst, March 19, 2003; 95(6): 470 - 478. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. M. Bossuyt, J. B. Reitsma, D. E. Bruns, C. A. Gatsonis, P. P. Glasziou, L. M. Irwig, D. Moher, D. Rennie, H. C.W. de Vet, and J. G. Lijmer The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration Ann Intern Med, January 7, 2003; 138(1): W1 - W12. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. M. Bossuyt, J. B. Reitsma, D. E. Bruns, C. A. Gatsonis, P. P. Glasziou, L. M. Irwig, D. Moher, D. Rennie, H. C.W. de Vet, and J. G. Lijmer The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration Clin. Chem., January 1, 2003; 49(1): 7 - 18. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 Performance of a Statistical Model to Predict Stroke Outcome in the Context of a Large, Simple, Randomized, Controlled Trial of Feeding Stroke, January 1, 2003; 34(1): 127 - 133. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. May Commentary: Improved coronary risk prediction using neural networks Int. J. Epidemiol., December 1, 2002; 31(6): 1262 - 1264. [Full Text]
|
|
|
|
|
|
B. P. Tabaei and W. H. Herman A Multivariate Logistic Regression Equation to Screen for Diabetes : Development and validation Diabetes Care, November 1, 2002; 25(11): 1999 - 2003. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. McClellan Overview: As to Diseases, Make a Habit of Two Things -- To Help, or at Least Do No Harm J. Am. Soc. Nephrol., November 1, 2002; 13(11): 2817 - 2819. [Full Text] [PDF]
|
|
|
|
|
|
J. F. Piccirillo, P. D. Lacy, A. Basu, and E. L. Spitznagel Development of a New Head and Neck Cancer-Specific Comorbidity Index Arch Otolaryngol Head Neck Surg, October 1, 2002; 128(10): 1172 - 1179. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. A. van Klei, K. G. M. Moons, A. T. Rheineck-Leyssius, C. J. Kalkman, C. L. G. Rutten, J. T. A. Knape, and D. E. Grobbee Validation of a clinical prediction rule to reduce preoperative type and screen procedures{dagger}{dagger}{dagger} Br. J. Anaesth., August 1, 2002; 89(2): 221 - 225. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. Brindle and M. May Commentary: The prediction of coronary heart disease risk in individuals-- an imprecise science Int. J. Epidemiol., August 1, 2002; 31(4): 822 - 824. [Full Text] [PDF]
|
|
|
|
|
|
M. Graefen, P. I. Karakiewicz, I. Cagiannos, D. I. Quinn, S. M. Henshall, J. J. Grygiel, R. L. Sutherland, P. D. Stricker, E. Klein, P. Kupelian, et al. International Validation of a Preoperative Nomogram for Prostate Cancer Recurrence After Radical Prostatectomy J. Clin. Oncol., August 1, 2002; 20(15): 3206 - 3212. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. S. Glas, B. A.C.M. Pijnenburg, J. G. Lijmer, K. Bogaard, M. de Roos, J. N. Keeman, R. M.J.M. Butzelaar, and P. M.M. Bossuyt Comparison of diagnostic decision rules and structured data collection in assessment of acute ankle injury Can. Med. Assoc. J., March 1, 2002; 166(6): 727 - 733. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Graefen, P. I. Karakiewicz, I. Cagiannos, E. Klein, P. A. Kupelian, D. I. Quinn, S. M. Henshall, J. J. Grygiel, R. L. Sutherland, P. D. Stricker, et al. Validation Study of the Accuracy of a Postoperative Nomogram for Recurrence After Radical Prostatectomy for Localized Prostate Cancer J. Clin. Oncol., February 15, 2002; 20(4): 951 - 956. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. B. Bach, D. Schrag, D. M. Nierman, D. Horak, P. White Jr, J. W. Young, and J. S. Groeger Identification of poor prognostic features among patients requiring mechanical ventilation after hematopoietic stem cell transplantation Blood, December 1, 2001; 98(12): 3234 - 3240. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R Marchioli, F Avanzini, F Barzi, C Chieffo, A Di Castelnuovo, M.G Franzosi, E Geraci, A.P Maggioni, R.M Marfisi, N Mininni, et al. Assessment of absolute risk of death after myocardial infarction by use of multiple-risk-factor assessment equations; GISSI-Prevenzione mortality risk chart Eur. Heart J., November 2, 2001; 22(22): 2085 - 2103. [Abstract] [PDF]
|
|
|
|
|
|
D. I. Quinn, S. M. Henshall, A.-M. Haynes, P. C. Brenner, R. Kooner, D. Golovsky, J. Mathews, G. F. O'Neill, J. J. Turner, W. Delprado, et al. Prognostic Significance of Pathologic Features in Localized Prostate Cancer Treated With Radical Prostatectomy: Implications for Staging Systems and Predictive Models J. Clin. Oncol., August 15, 2001; 19(16): 3692 - 3705. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. C. Walter, R. J. Brand, S. R. Counsell, R. M. Palmer, C. S. Landefeld, R. H. Fortinsky, and K. E. Covinsky Development and Validation of a Prognostic Index for 1-Year Mortality in Older Adults After Hospitalization JAMA, June 20, 2001; 285(23): 2987 - 2994. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. U. Weir, P. A. G. Sandercock, S. C. Lewis, D. F. Signorini, and C. P. Warlow Variations Between Countries in Outcome After Stroke in the International Stroke Trial (IST) Stroke, June 1, 2001; 32(6): 1370 - 1377. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. C. McCrory, C. Brown, S. E. Gelfand, and P. B. Bach Management of Acute Exacerbations of COPD : A Summary and Appraisal of Published Evidence Chest, April 1, 2001; 119(4): 1190 - 1209. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K. E. Covinsky, E. Kahana, B. Kahana, K. Kercher, J. G. Schumacher, and A. C. Justice History and Mobility Exam Index to Identify Community-Dwelling Elderly Persons at Risk of Falling J. Gerontol. A Biol. Sci. Med. Sci., April 1, 2001; 56(4): 253M - 259. [Abstract] [Full Text]
|
|
|
|
 |
|
 A. WADE Derivation versus validation Arch. Dis. Child., December 1, 2000; 83(6): 459 - 460. [Full Text]
|
|
|
|
|
|
J. Klastersky, M. Paesmans, E. B. Rubenstein, M. Boyer, L. Elting, R. Feld, J. Gallagher, J. Herrstedt, B. Rapoport, K. Rolston, et al. The Multinational Association for Supportive Care in Cancer Risk Index: A Multinational Scoring System for Identifying Low-Risk Febrile Neutropenic Cancer Patients J. Clin. Oncol., August 16, 2000; 18(16): 3038 - 3051. [Abstract] |
Article
