 |
 |
|
Home |
Current Issue |
Past Issues |
In the Clinic |
ACP Journal Club |
CME |
Collections |
Audio/Video |
Mobile |
Subscribe |
Tools |
Help |
ACP Online
|
Rapid Responses to:
|
|
Electronic letters published:
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif){kind=icon}
Improving the Quality of Evaluations of Quality Improvement
Improving Improvement
In Response
|
|
|||
|
Bruce Agins, MD, MPH NY State Dept. of Health, Clemens M. Steinbock, MD
Send rapid response to journal:
bda01{at}health.state.ny.us Bruce Agins, et al.
|
The landmark study by Landon et. al. is a rigorous evaluation of a major HIV quality improvement (QI) initiative and of large-scale QI collaboratives . Specific factors pertinent to the national network of HIV services in the US temper the findings of this important study, particularly with respect to comparison of results between intervention and control clinics. Although the authors carefully mention that results may be contaminated because control clinics participated in another QI program, they suggest that this intervention occurred prior to the collaborative, when in fact, it continued during the collaborative, and is ongoing. The National HIVQUAL Project, also funded by the HIV/AIDS Bureau of HRSA, is another resource for QI grantees managed by the NYS Department of Health AIDS Institute providing onsite consultation to promote development of quality management programs for these clinics. In spite of acknowledging this potential ‘contamination’ the analysis does not consider the effect of this parallel QI consultation among both control and intervention participants. Possibly, differences in results between those clinics would have differed from those which did not participate in either. Another critical issue is the analysis of findings within a participating agency. Because selection of patients for testing changes in the collaborative included bias through the choice of a population of focus, the authors reviewed representing this group as well as those not in this group. They do not however describe differences in results between these groups, leaving open the question of whether interventions produced improvements in the testing group, as opposed to the entire group, which would support positive benefit for patients targeted by QI strategies, identifying instead a potential problem with spread of these changes throughout the care system from those achieved in the selected group. Targeting diffusion of change is a specific strategy that could become integrated into the collaborative framework. The evaluation of QI collaboratives poses substantial challenges given the multitude of changes occurring simultaneously and the existence of concurrent external and internal stimuli to improve care. This study highlights important issues in the evaluation methodology of QI initiatives - which needs to be refined before we dismiss the collaborative model for improvement. With adherence to its own fundamental principles, the collaborative model can be improved to enhance its quality and our understanding to further the science of improvement. Bruce Agins, MD, MPH Medical Director, AIDS Institute New York State Dept. of Health Phone: 212.268.6142 Fax: 212.268.6098 Address: NYSDOH AIDS Institute 5 Penn Plaza; First Floor New York, NY 10001 Conflict of Interest:None declared |
|||
|
|
|||
|
Paul Batalden, MD Dartmouth Medical School
Send rapid response to journal:
Paul.Batalden{at}Dartmouth.edu Paul Batalden
|
Dear Editor, The recent articles by Landon, et al and Mittman are welcome encouragement in the effort to improve the improvement of clinical care. As they suggest, published studies with greater capacity to discern what has happened and greater ability to describe the context will be helpful. Let me extend their suggestions for future published reports: 1. About the populations. Patients with HIV AIDS fit into several subpopulations, e.g., drug-using/not, transfusion or blood products- acquired/not, etc. Improving care for subgroups may require different interventions. Were the efforts in either control or intervention group stratified and designed with the subgroup needs in mind? What knowledge about ‘desirable care’ did the patients within a site have? What patterns of patient involvement in the continuing redesign and improvement of the care was the local habit/practice in the intervention and control settings? 2. About the changes introduced. Did the changes introduced match the known local barriers to better performance? Were they designed to fit the particular local situation? Was an ongoing process ‘independent of the intervention’ of local problem identification and remediation underway in either group? 3. About the setting’s change history. What was the change history in the settings? Did the changes arise from intact care units working together to make change, or were their changes done in response to a charismatic leader? Was there a clinical champion of the change? Did the intervention build on prior successful changes? If the practices didn’t change, did the provider knowledge change? Was the length of the follow- up appropriate to assess the effects? Was there a curious local leader with demonstrated change skill in either setting? 4. About the use of measurement. What were the local habits about using outcome and process data for caregiving and for change? Did the interventions build on those patterns? Were the available data relevant for units that had to change? How did local practice environments encourage the linkage of data to care and its improvement by the clinical microsystems? Were local microsystems locally accountable for outcomes? 5. About the setting’s self-awareness. Did the caregivers understand themselves to be part of an interdependent ‘system’ and did they function that way as they considered changes? How did they understand their own work processes of HAART therapy and the other desired clinical outcomes such as immunization? Did they know their own processes well enough to connect measures of the variation to their work? 6. About the community surrounding the settings. What were the community HIV AIDS practices in the small area surrounding the settings? Did the sites intentionally adopt or differentiate themselves from those patterns? If so, why? Did the interventional design build on that knowledge? Was there geographic variation in the available health resources for the sites? These and other evaluative studies with even greater detail can help us build a solid empirical base under efforts to change and improve clinical care. Sincerely, Paul Batalden, M.D. Center for the Evaluative Clinical Sciences Dartmouth Medical School Landon, et al. Effects of a Quality Improvement Collaborative on the Outcome of Care of Patients with HIV Infection:the EQHIV Study. Ann Int Med 2004,140:887-896. Mittman,B.S. Creating the Evidence Base for Quality Improvement Collaboratives. Ann Int Med 2004,140:897-901 Conflict of Interest:None declared |
|||
|
|
|||
|
Bruce E. Landon, M.D., M.B.A., M.Sc. Harvard Medical School, Ira B. Wilson, Paul D. Cleary
Send rapid response to journal:
landon{at}hcp.med.harvard.edu Bruce E. Landon, et al.
|
In response: Dr. Batalden suggests that more information than is provided in our article1 is needed to understand how to improve quality improvement (QI) interventions. We agree completely. A good evaluation study should first assess whether an intervention was successful.2 When there is no effect, a good study should provide information about how well the program was implemented, as well as information on potential moderating effects. Annals articles must be concise and we could not include all of these details in our article. However, we did assess virtually all of the factors mentioned by Dr. Batalden. We conducted detailed assessments of the almost 1,500 change initiatives attempted.1 We also surveyed clinicians and medical directors at each of the clinics both before and after the intervention to assess these factors, and we made site visits to a sample of clinics to collect qualitative data that might shed light on why the intervention was not more successful. Prior to publishing our evaluation, we conducted preliminary analyses of all that information to assure ourselves that we were not misrepresenting the results or missing a critical determinant of success. We hope that subsequent analyses will improve our understanding of why the intervention described was not more successful. Dr. Agins is concerned that our evaluation might have been contaminated by the participation of control clinics in the HIV QUAL program or other quality improvement efforts. We had the same concern and asked each study clinic detailed questions about their participation in QI activities both before and after the collaborative. We did not find much evidence of ongoing quality improvement activities in the control clinics. The relatively small improvement in both the intervention and control sites suggests that our lack of results was not due to large improvements in the control clinics. We did assess whether there were larger effects in the “population of focus” in the 11 clinics (out of 44 total) that restricted their efforts to a subset of the clinic population.1 In models that compared improvement in the population of focus to control clinics, only the increase in the number of visits in 3 or 4 quarters was significant (p=.04). We hope that future studies continue to address these types of issues to enhance our understanding of the interventions that will lead to the greatest improvements. Bruce E. Landon, M.D., M.B.A, M.Sc., Paul D. Cleary, Ph.D. Harvard Medical School Boston, MA 02115 Ira B. Wilson, M.D., M.Sc. Tufts-New England Medical Center Boston, MA 02111 1. Landon BE, Wilson IB, McInnes K, et al. Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study. Ann Intern Med. 2004;140:887-896. 2. Rossi PH, Freeman HE, Lipsey MW. Evaluation. A Systematic Approach. Sixth ed. Thousand Oaks: SAGE Publications; 1999. Conflict of Interest:None declared |
|||
