Effects of Training in Direct Observation of Medical Residents' Clinical Competence

A Randomized Trial

1 June 2004 | Volume 140 Issue 11 | Pages 874-881

Background: Faculty observation of residents and students performing clinical skills is essential for reliable and valid evaluation of trainees.

Objective: To evaluate the efficacy of a new multifaceted method of faculty development called direct observation of competence training.

Design: Controlled trial of faculty from 16 internal medicine residency programs using a cluster randomization design.

Setting: Academic medical centers.

Participants: 40 internal medicine teaching faculty members: 17 in the intervention group and 23 in the control group.

Measurements: Changes in faculty comfort performing direct observation, faculty satisfaction with workshop, and changes in faculty rating behaviors 8 months after completing the training.

Intervention: The direct observation of competence workshop combines didactic mini-lectures, interactive small group and videotape evaluation exercises, and evaluation skill practice with standardized residents and patients.

Results: 37 faculty members (16 in the intervention group and 21 in the control group) completed the study. Most of the faculty in the intervention group (14 [88%]) reported that they felt significantly more comfortable performing direct observation compared with control group faculty (4 [19%]) (P = 0.04), and all intervention faculty rated the training as outstanding. For 9 videotaped clinical encounters, intervention group faculty were more stringent than controls in their evaluations of medical interviewing, physical examination, and counseling; differences in ratings for medical interviewing and physical examination remained statistically significant even after adjustment for baseline rating behavior.

Limitations: The study involved a limited number of residency programs, and faculty did not rate the performance of actual residents.

Conclusion: Direct observation of competence training, a new multifaceted approach to faculty development, leads to meaningful changes in rating behaviors and in faculty comfort with evaluation of clinical skills.

Editors' Notes Context Reliable methods for evaluating clinical competence of medical trainees are needed. Contribution This cluster randomized trial involving 16 internal medicine programs evaluated a 4-day course that taught faculty direct observation methods for evaluating clinical competence. Eight months later, course participants reported greater comfort with direct observation to evaluate residents and rated videotaped clinical encounters between standardized residents and patients more stringently than did faculty not taking the course. Cautions Larger studies that use ratings of actual residents and that include resident feedback are needed to establish the transportability and efficacy of the program. –The Editors

 

Author and Article Information

From Yale University, New Haven, Connecticut, and the Uniformed Services University of the Health Sciences, Bethesda, Maryland.

Note: Dr. Hawkins was Associate Professor of Medicine at the Uniformed Services University of the Health Sciences during the duration of this study.

Acknowledgments: The authors thank Yun Wang, MS, for his statistical expertise, Leslie Galaty for her help in data entry and organization, Dr. Jeffrey Chung for his invaluable assistance in the research, Dr. Louis Pangaro for his assistance in reviewing videotapes, Dr. John Norcini and Rebecca Lipner for their statistical advice, and Drs. Patrick O'Connor and Daniel Duffy for their review of the research.

Grant Support: By the Robert Wood Johnson Foundation and the American Board of Internal Medicine Foundation.

Potential Financial Conflicts of Interest:Honoraria: E.S. Holmboe (American Board of Internal Medicine Foundation); Grants received: E.S. Holmboe (Robert Wood Johnson Foundation, American Board of Internal Medicine, Bureau of Primary Healthcare, Agency for Healthcare Research and Quality, Mitchell Center for Prisoner of War Studies, Reynolds Foundation, American College of Physicians Foundation).

Corresponding Author: Eric S. Holmboe, MD, Department of Medicine, Waterbury Hospital, Pomeroy 6, 64 Robbins Street, Waterbury, CT 06721.

Current Author Addresses: Dr. Holmboe: Department of Medicine, Waterbury Hospital, Pomeroy 6, 64 Robbins Street, Waterbury, CT 06721.

Dr. Hawkins: National Board of Medical Examiners, 3750 Market Street, Philadelphia, PA 19104-3102.

Dr. Huot: Yale University School of Medicine, 33 Cedar Street, 1074 LMP, New Haven, CT 06520.

Author Contributions: Conception and design: E.S. Holmboe, R.E. Hawkins, S.J. Huot.

Analysis and interpretation of the data: E.S. Holmboe, R.E. Hawkins.

Drafting of the article: E.S. Holmboe, S.J. Huot.

Critical revision of the article for important intellectual content: E.S. Holmboe, R.E. Hawkins.

Final approval of the article: E.S. Holmboe, R.E. Hawkins, S.J. Huot.

Provision of study materials or patients: E.S. Holmboe, R.E. Hawkins.

Obtaining of funding: E.S. Holmboe.

Administrative, technical, or logistic support: E.S. Holmboe, R.E. Hawkins.

Collection and assembly of data: E.S. Holmboe, R.E. Hawkins.