Analyzing Computer-based Decision Support Systems

TO THE EDITOR:

The careful analysis of computer-based decision making by Johnson and colleagues [1] is an important contribution. Although we agree with most of their conclusions, we disagree strongly with two of their criteria for methodologic adequacy.

First, they imply by their scoring system that randomization by practice or institution (rather than by patient or physician) is always preferred. Randomization in clinical trials is designed to control bias. Studies of computer-based decision support systems typically require hierarchical, clustered designs with the intervention applied to the physician (computer reminders) and the outcomes measured in patients (for example, delivery of preventive care, cost of care, and blood pressure control). Randomization by patient provides the most efficient use of data from a fixed number of patients; randomization by physician, the next most efficient; and randomization by practice, the least efficient.

Randomizing by patient or provider does increase the risk that the intervention will contaminate the control cases and reduce the chance of significant observed differences between intervention and controls; however, contamination is often weaker than expected. We have observed large effects even when physicians served as their own controls in crossover studies [2]. Furthermore, some interventions (for example, the presentation of results of complex kinetic models) are almost immune to contamination because the provider cannot carry out the intervention in the control state without the help of the computer.

Requiring that separate clinics or institutions be the unit of randomization reduces the risk for contamination but presents its own problems. If only a few clinics or practices are available, observed effects may be attributable to differences in the clinic sites, not the intervention. Obtaining many truly independent sites can be impossible (or hopelessly expensive), especially when it is necessary to integrate the intervention into institutional computer systems. Thus, the preferred unit of randomization depends on the nature of the intervention, the organization of the practice, and the hypothesis studied.

Whatever the randomization method used, the analysis must address the clustering effects within providers or institutions while allowing for the correlated outcomes of patients within each provider or practice. Several techniques [3, 4]—maximum likelihood models, quasi-likelihood models, and hierarchical Bayesian models—are now available for such analyses.

• Copyright ©2004 by the American College of Physicians