A Randomized Trial of Ways To Describe Test Accuracy: The Effect on Physicians' Post-Test Probability Estimates

2 August 2005 | Volume 143 Issue 3 | Pages 184-189

Background: Some people believe that likelihood ratios provide diagnostic information that is more useful than sensitivity and specificity estimates.

Objective: To assess how physicians' estimates about probability of illness are affected by the presentation of a diagnostic test's value as an estimate of sensitivity and specificity versus a likelihood ratio or an inexact numerical graphic.

Design: Random assignment of vignettes with different presentation formats of diagnostic test accuracy.

Setting: Auditorium at a continuing medical education conference.

Participants: 183 physicians.

Intervention: After estimating probabilities of 6 common illnesses described in patient vignettes, physicians reviewed pertinent test results presented in 1 of 3 formats.

Measurements: Physicians' probability estimates of illness before and after receiving test information, and post-test probability estimates based on the Bayes theorem.

Results: Absolute percentage point differences between the physicians' estimated and the Bayes-based post-test probabilities varied from –7 to 31, from –7 to 28, and from 1 to 29 for the sensitivity and specificity, likelihood ratio, and graphical groups, respectively. Mean differences of probability estimates between the sensitivity and specificity and the likelihood ratio groups were small for all vignettes (–2 to 3 percentage points; summary mean z value across the 6 vignettes, 0.04 [95% CI, –0.14 to 0.21]).

Limitations: The small pool of participants (who were potentially selected) and the limited number of vignettes prevented a more detailed analysis of relationships between the interpreted strength of diagnostic evidence and estimations of illness probability.

Conclusions: These findings suggest that presenting diagnostic test accuracy with likelihood ratios does not affect some physicians' estimates of illness probability compared with presenting diagnostic test results as sensitivity and specificity.

Editors' Notes Context There are several ways to present information about the accuracy of diagnostic tests. Contribution In this vignette-based study, 183 physicians read 6 scenarios that included patients' presenting symptoms, risk profiles, and test results. Physicians were randomly assigned to view information about test accuracy as sensitivity and specificity, a likelihood ratio, or an inexact graphic of the likelihood ratio. In general, the groups estimated similar post-test probabilities of disease regardless of how diagnostic accuracy was presented. Implications This preliminary work suggests some physicians may interpret sensitivity and specificity and likelihood ratio information in a similar manner. –The Editors

 

Author and Article Information

From the University of Zurich, Zurich, Switzerland.

Acknowledgments: The authors thank Dr. Otto Brändli for the opportunity to conduct this study during the congress of the Zurich Lung League in Davos, Switzerland.

Grant Support: By the Helmut Horten Foundation. Dr. Bachmann's work was supported by the Swiss National Science Foundation (grants 3233B0-103182 and 3200B0-103183).

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: Milo A. Puhan, MD, Horten Centre, University Hospital of Zurich, Postfach Nord, 8091 Zurich, Switzerland; e-mail, milo.puhan{at}evimed.ch.

Current Author Addresses: Drs. Puhan and Steurer: Horten Centre, University Hospital of Zurich, Postfach Nord, 8091 Zurich, Switzerland.

Dr. Bachmann: University of Berne, Department of Social and Preventive Medicine, Finkenhubelweg 11, 3012 Berne, Switzerland.

Dr. ter Riet: Academic Medical Center, Deptment of General Practice, Room J2-118, 1105 AZ Amsterdam, the Netherlands.

Author Contributions: Conception and design: M.A. Puhan, J. Steurer, G. ter Riet.

Analysis and interpretation of the data: M.A. Puhan, L.M. Bachmann, J. Steurer, G. ter Riet.

Drafting of the article: M.A. Puhan.

Critical revision of the article for important intellectual content: L.M. Bachmann, J. Steurer, G. ter Riet.

Final approval of the article: M.A. Puhan, L.M. Bachmann, J. Steurer, G. ter Riet.

Provision of study materials or patients: M.A. Puhan.

Statistical expertise: M.A. Puhan, G. ter Riet.

Collection and assembly of data: M.A. Puhan.

Read all Rapid Responses

Against the odds

Daniel J. Brotman

Annals Online, 9 Aug 2005 [Full text]

Almost no room for improvement!

Mike Broce, et al.

Annals Online, 18 Aug 2005 [Full text]

Reply to letters

Milo A. Puhan, et al.

Annals Online, 3 Oct 2005 [Full text]