RSS Feeds
-
Response to Dr. Liu and colleagues
Submit responseWe thank Dr. Liu and colleagues for their comments regarding our recent article. They made several statements about our results that readers less familiar with the article may find misleading. While both groups showed improvements in many aspects of antimicrobial prophylaxis, there were no significant differences in the extent of improvement between the two groups on any indicator. Our conclusion is correct in this regard. Our study was designed to examine the absolute difference in change between the two groups not the relative difference in change, thus the proper statistic to evaluate the effect of participation in the collaborative group for the “all-or-none” indicator shows a 6.3% greater improvement in the active intervention (95% CI, -7.3%,19.8%).
We agree that improvement interventions are complex social processes that should be theory-based and explained in context using evaluation models drawn from social science (1, 2). However, we disagree with the statement that the real value of this study appears to lie in the discovery of the heterogeneity and the opportunity it offers for explaining it. Participants in all randomized studies, be they patients or organizations, exhibit heterogeneity. One of the greatest values of a randomized trial design is the ability to answer the question of effectiveness in the face of heterogeneity, thereby addressing the expectation of improvement for the next patient (or organization) that adopts the evaluated treatment. We agree that case studies of success can be extremely helpful in stimulating quality improvement, but the fact that we do not know which mechanisms worked within which circumstances does not negate the value of the cluster randomized trial as an evaluative methodology.
There is much to be done to improve the quality of health care delivery and quality improvement collaboratives have a prominent role. In a recent systematic review of the impact of quality improvement collaboratives, Schouten et al. concluded that the evidence of their effectiveness is encouraging but limited; they called for studies with a balance of both process-oriented reports and rigorously controlled designs to understand why some collaboratives succeed while others have little effect on practice (3). Nevertheless, quality improvement collaboratives can be expensive to implement and tightening resources require the selection of cost-effective strategies. Data from rigorous evaluations such as that collected for the TRAPE study are needed to help organizations select among potential improvement strategies.
References
Grol R, Grimshaw J From best evidence to best practice. Lancet. 2003;11;362(9391):1225-30.
Pawson R, Tilley N. Realistic Evaluation. London, England: Sage Publications Ltd: 1997.
Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol R. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ 2008;336:1491-1494
Conflict of Interest:
None declared
-
suggestions to the authors
Submit responseResponse to “The Effect of a Quality Improvement Collaborative to Improve Antimicrobial Prophylaxis in Surgical Patients”
Antimicrobial prophylaxis is important to control the occurrence of postoperative infection caused by surgical site, serving therefore to delay or eliminate microbial growth. For that purpose, care in the selection, timing and duration of antibiotic is necessary as a way to reduce morbidity associated with invasive procedures.
Kritchevsky SB et al. evaluated the effects of a quality improvement collaborative on preoperative antimicrobial prophylaxis in hospitals that received support from a group of specialists versus hospitals that did not receive such assistance.
However, we noticed some methodological issues that might have affected the results. The volunteer bias, usually detected among human being, was also present among the clustered hospitals enrolled in the study. In this case, the entire studied population was more willing to change and to improve the quality of antibiotic prophylaxis. The baseline data confirmed that the performance of the intervention and control groups was already high for most of the variables related to the preoperative antimicrobial prophylaxis. So, there was not enough room for improvement in the re-measurement phase. Moreover, the feedback report was a stimulus sufficient to promote change in timing, duration, but there was no meaningful increase in the use of a single preoperative dose of antimicrobials.
Why the investigators did not use a more robust outcome, such as the rate of postoperative infection? It would be easy to standardize a protocol and measure a direct outcome. The final target should be the reduction in the infection rate by minimizing the mistakes in the preoperative antimicrobial prophylaxis.
Conflict of Interest:
None declared
-
Do Quality Improvement Collaboratives Improve Antimicrobial Prophylaxis in Surgical Patients?
Submit responseTO THE Editor: The conclusions of the article by Kritchevsky et al (1) stated in the Abstract do not accurately represent the study’s findings. In fact, there is substantial reason to conclude that the collaborative was of meaningful clinical value. For example, improvement was statistically significant in 4 of the 6 mean performance measures in the intervention group but in only 3 of 6 improved in the feedback-only (control) group. Moreover, improvement in the mean “all or none” measure was 15.6% greater in the intervention than the control group, which actually exceeds the 15% difference sought in the power calculation for the primary outcome. Although that particular observed difference was not statistically significant, the upper bound of the confidence interval for the all-or-none measure does not exclude a between-group advantage for the intervention group as great as 49%.
Although the article did not include the changes observed in each hospital, the wide confidence intervals for many of the pre-post measures (for example, 15.6 percentage points for the primary outcome in the control group, which is 20.8% of the baseline rate) indicate substantial heterogeneity of response to both interventions among individual hospitals, including quite large responses in some hospitals. Heterogeneity of response to either biological or social interventions is not simply “noise” (2); rather it serves as the starting point for understanding the mechanisms involved in those responses. The real value of this study therefore appears to lie in the discovery of that heterogeneity and the opportunity it offers for explaining it – why feedback alone or feedback-plus-collaborative was effective, for whom, and in what circumstances (3) – as much as in the demonstration of whether, in the aggregate, each intervention “worked.” Numbers are not explanations (4), so it is fortunate that the authors have pinpointed in the discussion a number of local and external factors that could explain the differential impact of the interventions within each study group, for example, diversity of staff training and experience, variation in total and type of staff participation, and variable involvement of institutional leadership.
Unfortunately, however, the study was not designed to explain how the best individual hospital performers differed from the worst and why. For example, we have no way of knowing whether the history of successful improvement efforts differed among participating hospitals; or exactly which staff participated in the collaborative and why, and how were they involved in the care processes being changed.
Changing healthcare work is a complex, context-bound social process. Evaluation of any improvement intervention therefore requires clarity regarding the theory that underlies its selection; recognition of the contributions of individual stakeholders to its effects; awareness of the sequence and execution of its individual steps; recognition of the power relationships among people involved in its implementation; understanding of the multiple realities that affect its execution: timing, culture, resource allocation, staffing, competing priorities; awareness that most interventions start by being adapted to fit local circumstances, and that the interventions and the contexts in which they are implemented evolve over time in response to the resulting changes. For their findings to be meaningful, future studies of improvement interventions will need to include these elements in their study designs (5).
References
1) Kritchevsky SB, Braun BI, Bush AJ, Bozikis MR, Kusek L, Burke JP, et al. The effect of a quality improvement collaborative to improve antimicrobial prophylaxis in surgical patients. A randomized trial. Ann Intern Med 2008;149:472-08.
2) Heng HHQ. The conflict between complex systems and reductionism. JAMA 2008;300:1580-1.
3) Pawson R, Greenhalgh T, Haravey G, Walshe K. Realist review – a new method of systematic review designed for complex policy interventions. J Health Services Res & Policy. 2005;10;21-34.
4) Vandenbroucke JP. Observational research, randomised trials, and two views of medical science. PLoS Medicine 2008;5:e67.
5) Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S. Publication guidelines for improvement studies in health care: evaluation of the SQUIRE project. Ann Intern Med (in press).
Conflict of Interest:
None declared