Incorporating Quality of Evidence into Decision Analytic Modeling

16 January 2007 | Volume 146 Issue 2 | Pages 133-141

Our objective was to illustrate the effects of using stricter standards for the quality of evidence used in decision analytic modeling. We created a simple 10-parameter probabilistic Markov model to estimate the cost-effectiveness of directly observed therapy (DOT) for individuals with newly diagnosed HIV infection. We evaluated quality of evidence on the basis of U.S. Preventive Services Task Force methods, which specified 3 separate domains: study design, internal validity, and external validity. We varied the evidence criteria for each of these domains individually and collectively. We used published research as a source of data only if the quality of the research met specified criteria; otherwise, we specified the parameter by randomly choosing a number from a range within which every number has the same probability of being selected (a uniform distribution). When we did not eliminate poor-quality evidence, DOT improved health 99% of the time and cost less than $100 000 per additional quality-adjusted life-year (QALY) 85% of the time. The confidence ellipse was extremely narrow, suggesting high precision. When we used the most rigorous standards of evidence, we could use fewer than one fifth of the data sources, and DOT improved health only 49% of the time and cost less than $100 000 per additional QALY only 4% of the time. The confidence ellipse became much larger, showing that the results were less precise. We conclude that the results of decision modeling may vary dramatically depending on the stringency of the criteria for selecting evidence to use in the model.

Author and Article Information

From Yale University School of Medicine and Connecticut Veterans Affairs Healthcare System, New Haven, Connecticut, and University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania.

Grant Support: By grant K23 AA14483-01 from the National Institute of Alcohol Abuse and Alcoholism.

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: R. Scott Braithwaite, MD, MSc, Yale University School of Medicine, Veterans Affairs Connecticut Healthcare System, Mailstop 11 ACSLG, 950 Campbell Avenue, West Haven, CT 06516; e-mail, ronald.braithwaite{at}va.gov.

Current Author Addresses: Drs. Braithwaite and Justice: Yale University School of Medicine, Veterans Affairs Connecticut Healthcare System, Mailtop 11 ACSLG, 950 Campbell Avenue, West Haven, CT 06516

Dr. Roberts: University of Pittsburgh School of Medicine, 200 Meyran Avenue, 2nd Floor, Pittsburgh, PA 15213.