 |
 |
|
16 November 2004 | Volume 141 Issue 10 | Pages 771-780
Background: Worldwide efforts are under way to improve the quality of clinical practice. Most quality measurements, however, are poorly validated, expensive, and difficult to compare among sites.
Objective: To validate whether vignettes accurately measure the quality of clinical practice by using a comparison with standardized patients (the gold standard method), and to determine whether vignettes are a more or less accurate method than medical record abstraction.
Design: Prospective, multisite study.
Setting: Outpatient primary care clinics in 2 Veterans Affairs medical centers and 2 large, private medical centers.
Participants: 144 of 163 eligible physicians agreed to participate, and, of these, 116 were randomly selected to see standardized patients, to complete vignettes, or both.
Measurements: Scores, expressed as the percentage of explicit quality criteria correctly completed, were obtained by using 3 methods.
Results: Among all physicians, the quality of clinical practice as measured by the standardized patients was 73% correct (95% CI, 72.1% to 73.4%). By using exactly the same criteria, physicians scored 68% (CI, 67.9% to 68.9%) when measured by the vignettes but only 63% (CI, 62.7% to 64.0%) when assessed by medical record abstraction. These findings were consistent across all diseases and were independent of case complexity or physician training level. Vignettes also accurately measured unnecessary care. Finally, vignettes seem to capture the range in the quality of clinical practice among physicians within a site.
Limitations: Despite finding variation in the quality of clinical practice, we did not determine whether poorer quality translated into worse health status for patients. In addition, the quality scores are based on measurements from 1 patient–provider interaction. As with all other scoring criteria, vignette criteria must be regularly updated.
Conclusions: Vignettes are a valid tool for measuring the quality of clinical practice. They can be used for diverse clinical settings, diseases, physician types, and situations in which case-mix variation is a concern. They are inexpensive and easy to use. Vignettes are particularly useful for comparing quality among and within sites and may be useful for longitudinal evaluations of interventions intended to change clinical practice.
Author and Article Information
From San Francisco Veterans Affairs Medical Center, Institute for Global Health, University of California, San Francisco, and California Pacific Medical Center, San Francisco, California; University of California, Los Angeles, Veterans Affairs Greater Los Angeles Healthcare System, and Permanente Medical Group, Los Angeles, California; RAND, Santa Monica, California; and Veterans Affairs Center for the Study of Healthcare Provider Behavior, Sepulveda, California.
Acknowledgments: The authors thank Elizabeth O'Gara (University of California, Los Angeles), Julianne Arnall (Stanford University), and Molly Bates Efrusy and Ojig Yeretsian (University of California, San Francisco), who trained and coordinated the standardized patient visits; the staff of the hospitals that worked so diligently to simulate the visits, provide their laboratory data, and introduce the simulated medical records; Bret Lewis, Ed La Calle, and Dan Bertenthal for their programming assistance and keen insights; Anne Sunderland and Miriam Polon for their assistance with the manuscript; and the actors for their fine performances.
Grant Support: By grant 11R 98118-1 from the Veterans Affairs Health Service Research and Development Service, Washington DC. Dr. Peabody is also a recipient of a Senior Research Associate Career Development Award from 1998 to 2001 from the Department of Veterans Affairs.
Potential Financial Conflicts of Interest: None disclosed.
Requests for Single Reprints: John W. Peabody, MD, PhD, Institute for Global Health, 74 New Montgomery, Suite 508, San Francisco, CA 94105; e-mail, peabody{at}psg.ucsf.edu.
Current Author Addresses: Dr. Peabody: Institute for Global Health, 74 New Montgomery Street, Suite 508, San Francisco, CA 94105.
Dr. Luck: Department of Health Services, University of California, Los Angeles, School of Public Health, PO Box 951772, Los Angeles, CA 90095-1772.
Mr. Glassman: Division of General Internal Medicine (111G), Veterans Affairs Greater Los Angeles, 11301 Wilshire Boulevard, Los Angeles, CA 90073.
Dr. Jain: Medical Service (111), Veterans Affairs Medical Center, 4150 Clement Street, San Francisco, CA 94121.
Dr. Hansen: 3801 Sacramento Street, Suite 309, San Francisco, CA 94118.
Dr. Spell: 4950 Sunset Boulevard, Los Angeles, CA 90027.
Dr. Lee: Center for the Study of Healthcare Provider Behavior, Veterans Affairs Medical Center (152), 16111 Plummer Street, Building 25, Sepulveda, CA 91343-2036. IMPROVING PATIENT CARE
Improving Patient Care is a special section within Annals supported in part by the U.S. Department of Health and Human Services (HHS) Agency for Healthcare Research and Quality (AHRQ). The opinions expressed in this article are those of the authors and do not represent the position or endorsement of AHRQ or HHS.
Measuring the Quality of Physician Practice by Using Clinical Vignettes: A Prospective Validation Study
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
Related articles in Annals:
This article has been cited by other articles:
|
|
 |
|
  N B A T Janssen, F J Oort, P Fockens, D L Willems, H C J M de Haes, and E M A Smets Under what conditions do patients want to be informed about their risk of a complication? A vignette study J. Med. Ethics, May 1, 2009; 35(5): 276 - 282. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 O. Solon, K. Woo, S. A Quimbo, R. Shimkhada, J. Florentino, and J. W Peabody A novel method for measuring health care system performance: experience from QIDS in the Philippines Health Policy Plan., May 1, 2009; 24(3): 167 - 174. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. R. Terrell, L. A. Beebe, J. N. George, S. K. Vesely, and J. W. Mold Referral of patients with thrombocytopenia from primary care clinicians to hematologists Blood, April 23, 2009; 113(17): 4126 - 4127. [Full Text] [PDF]
|
|
|
|
 |
|
 J. Arndt, M. Vess, C. R. Cox, J. L. Goldenberg, and S. Lagle The Psychosocial Effect of Thoughts of Personal Mortality on Cardiac Risk Assessment Med Decis Making, March 1, 2009; 29(2): 175 - 181. [Abstract] [PDF]
|
|
|
|
 |
|
 J. Harting, G. M. Rutten, S. T. Rutten, and S. P Kremers A Qualitative Application of the Diffusion of Innovations Theory to Examine Determinants of Guideline Adherence Among Physical Therapists Physical Therapy, March 1, 2009; 89(3): 221 - 232. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C M Sox, J N Doctor, T D Koepsell, and D A Christakis The influence of types of decision support on physicians' decision making Arch. Dis. Child., March 1, 2009; 94(3): 185 - 190. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 N. L. Keating, K. A. Stoeckert, M. M. Regan, L. DiGianni, and J. E. Garber Physicians' Experiences With BRCA1/2 Testing in Community Settings J. Clin. Oncol., December 10, 2008; 26(35): 5789 - 5796. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. M. Weston, C. N. Sciamanna, and D. B. Nash Evaluating Online Continuing Medical Education Seminars: Evidence for Improving Clinical Practices American Journal of Medical Quality, November 1, 2008; 23(6): 475 - 483. [Abstract] [PDF]
|
|
|
|
 |
|
 E O'Neill, R McNamee, R Agius, M Gittins, L Hussey, and S Turner The validity and reliability of diagnoses of work-related mental ill-health Occup. Environ. Med., November 1, 2008; 65(11): 726 - 731. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. E. Guerra, P. A. Gimotty, J. A. Shea, J. A. Pagan, J. S. Schwartz, and K. Armstrong Effect of Guidelines on Primary Care Physician Use of PSA Screening: Results from the Community Tracking Study Physician Survey Med Decis Making, September 1, 2008; 28(5): 681 - 689. [Abstract] [PDF]
|
|
|
|
|
|
O. Kostopoulou, J. Oudhoff, R. Nath, B. C. Delaney, C. W. Munro, C. Harries, and R. Holder Predictors of Diagnostic Accuracy and Safe Management in Difficult Diagnostic Problems in Family Medicine Med Decis Making, September 1, 2008; 28(5): 668 - 680. [Abstract] [PDF]
|
|
|
|
 |
|
 S. O. Okelo, C. M. Patino, K. A. Riekert, B. Merriman, A. Bilderback, N. N. Hansel, K. Thompson, J. Thompson, R. Quartey, C. S. Rand, et al. Patient Factors Used by Pediatricians to Assign Asthma Treatment Pediatrics, July 1, 2008; 122(1): e195 - e201. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. L. Keating, M. B. Landrum, C. N. Klabunde, R. H. Fletcher, S. O. Rogers, W. R. Doucette, D. Tisnado, S. Clauser, and K. L. Kahn Adjuvant Chemotherapy for Stage III Colon Cancer: Do Physicians Agree About the Importance of Patient Age and Comorbidity? J. Clin. Oncol., May 20, 2008; 26(15): 2532 - 2537. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. Sirovich, P. M. Gallagher, D. E. Wennberg, and E. S. Fisher Discretionary Decision Making By Primary Care Physicians And The Cost Of U.S. Health Care Health Aff., May 1, 2008; 27(3): 813 - 823. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. K. Mangione, R. B Lopopolo, N. P Neff, R. L Craik, and K. M Palombaro Interventions Used by Physical Therapists in Home Care for People After Hip Fracture Physical Therapy, February 1, 2008; 88(2): 199 - 210. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. J. Weiner, A. Schwartz, R. Yudkowsky, G. D. Schiff, F. M. Weaver, J. Goldberg, and K. B. Weiss Evaluating Physician Performance at Individualizing Care: A Pilot Study Tracking Contextual Errors in Medical Decision Making Med Decis Making, December 1, 2007; 27(6): 726 - 734. [Abstract] [PDF]
|
|
|
|
 |
|
 C. Dumoulin, N. Korner-Bitensky, and C. Tannenbaum Urinary Incontinence After Stroke: Identification, Assessment, and Intervention by Rehabilitation Professionals in Canada Stroke, October 1, 2007; 38(10): 2745 - 2751. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. W Peabody and A. Liu A cross-national comparison of the quality of clinical care using vignettes Health Policy Plan., September 1, 2007; 22(5): 294 - 302. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. L. Leonard and M. C. Masatu Variations In The Quality Of Care Accessible To Rural Communities In Tanzania Health Aff., May 1, 2007; 26(3): w380 - w392. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 L. Li, Z. Wu, Y. Zhao, C. Lin, R. Detels, and S. Wu Using case vignettes to measure HIV-related stigma among health professionals in China Int. J. Epidemiol., February 1, 2007; 36(1): 178 - 184. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 V. Boolchand, G. Olds, J. Singh, P. Singh, A. Chak, and G. S. Cooper Colorectal screening after polypectomy: a national survey study of primary care physicians. Ann Intern Med, November 7, 2006; 145(9): 654 - 659. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. M. Stahl, M. Grady, G. Santiago, and R. L. Davis Optimizing Outcomes in Psychopharmacology Continuing Medical Education (CME): Measuring Learning and Attitudes That May Predict Knowledge Translation into Clinical Practice Focus, September 1, 2006; 4(4): 487. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. L Green Evaluating evidence-based practice performance Evid. Based Med., August 1, 2006; 11(4): 99 - 101. [Full Text] [PDF]
|
|
|
|
 |
|
 C. M. Sox, T. D. Koepsell, J. N. Doctor, and D. A. Christakis Pediatricians' Clinical Decision Making: Results of 2 Randomized Controlled Trials of Test Performance Characteristics Arch Pediatr Adolesc Med, May 1, 2006; 160(5): 487 - 492. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. E. Sirovich, D. J. Gottlieb, H. G. Welch, and E. S. Fisher Variation in the Tendency of Primary Care Physicians to Intervene Arch Intern Med, October 24, 2005; 165(19): 2252 - 2256. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Veloski, S. Tai, A. S. Evans, and D. B. Nash Clinical Vignette-Based Surveys: A Tool for Assessing Physician Practice Variation American Journal of Medical Quality, May 1, 2005; 20(3): 151 - 157. [Abstract] [PDF]
|
|
|
|
 |
|
 J. Norcini and M B. Anderson What the educators are saying BMJ, February 19, 2005; 330(7488): 405 - 405. [Full Text] [PDF]
|
|
|
|
|
|
J. Norcini Back to the Future: Clinical Vignettes and the Measurement of Physician Performance Ann Intern Med, November 16, 2004; 141(10): 813 - 814. [Full Text] [PDF]
|
|
Article
