Predictive Models for Primary Caregivers: Risky Business?

By informing the opinions of health care providers, future decision support systems will improve clinical practice. Yet many practitioners still approach technologically based tools with unquestioning awe rather than by trying to understand the mechanisms, applicability, and limitations of the tools. The key question in evaluating computer-based decision support tools is whether those tools can augment the native abilities of health care providers during clinical practice, not how they function in isolation as “omniscient oracles” [1, 2]. Practitioners who lose sight of their own clinical judgment when using a decision support tool, instead of seeing the tool as part of a balanced overall approach, do a disservice to themselves and their patients [3].

In this issue, Selker and colleagues [4] describe a practical decision support tool that rapidly estimates the patient-specific utility of thrombolytic therapy for patients with acute myocardial infarction. Predictive tools facilitate the distillation of a large volume of clinical experience into a convenient form that can be carried to the bedside and applied in a directed manner to a single patient. A 1991 study [5] suggested that as many as one fourth of physicians' unmet needs for information involve tailoring general information from the biomedical literature to fit individual patients. Predictive tools of the type described by Selker and colleagues could reduce critical delays in the initiation of therapies whose value depends on early, short-term administration. Used appropriately, a tool that accurately predicts the outcome of thrombolytic therapy in patients with acute myocardial infarction might save thousands of lives annually in the United States and could substantially reduce the rate of infarction-related illness. Similar tools might help reduce societal resistance to the adoption of medical technological innovations [6].

However, the development and use of predictive instruments in primary care evokes a concern initially voiced in the era of William Harvey and Thomas Sydenham. Philosopher-scientist Francis Bacon wrote in 1620 [7] that

“It is the peculiar and perpetual error of the human intellect to be more moved and excited by affirmatives than by negatives; whereas it ought properly to hold itself indifferently disposed toward both alike. Indeed, in the establishment of any true axiom, the negative instance is the more forcible of the two.”

and

“The induction which proceeds by simple enumeration is childish; its conclusions are precarious and exposed to peril from a contradictory instance; and it generally decides on too small a number of facts, and on those only which are at hand. But the induction which is to be available for the discovery and demonstration of sciences and arts, must analyze nature by proper rejections and exclusions; and then, after a sufficient number of negatives, come to a conclusion on the affirmative instance.”

To satisfy Bacon, developers and users of evidence-based predictive tools should ask several questions. How representative are the patients that were used in developing a given predictive tool? Which patients are likely to be misclassified and subjected to unnecessary interventions by a particular tool? And what is the severity of the adverse effects that occur in patients receiving unnecessary interventions?

Selker and colleagues constructed their predictive tool by using data from 4911 patients who participated in 13 high-quality randomized, controlled trials evaluating thrombolytic therapy for acute myocardial infarction [5]. The entry criteria for the thrombolytic trials (and the predictive instrument) were age 35 to 75 years; acute chest pain of at least 20 minutes duration within 9 hours of onset; systolic blood pressure greater than 85 mm Hg (after volume repletion) but less than 190 mm Hg; ST-segment elevation of at least 1 mm in two or more contiguous electrocardiographic leads; and no easily identified contraindications to thrombolytic therapy (for example, recent stroke, surgery, or trauma).

The overwhelming majority of patients with acute myocardial infarction develop this condition as a result of atherosclerosis, which is the primary indication for thrombolytic therapy. However, several patients who have acute myocardial infarction secondary to undetected, uncommon disorders, including aortic dissection extending to involve coronary arteries, infectious endocarditis with coronary embolism, and coronary vasculitis (polyarteritis nodosa, Takayasu arteritis, Wegener granulomatosis, and others), would also meet the predictive tool's entry criteria (that is, they would have inappropriate true-positive entry criteria). Other patients with rare conditions that mimic acute myocardial infarction, such as prolonged pain from Prinzmetal variant angina, acute pericarditis, acute myocarditis, cardiomyopathy, and primary and secondary cardiac neoplasms, could also meet the entry criteria (that is, they could have false-positive entry criteria).

When given thrombolytic therapy, patients who are inappropriately classified (that is, incorrect true-positive and false-positive entry criteria) would in general have intracranial hemorrhages and other adverse events at the baseline rates for the persons enrolled in the 13 large trials. However, for patients with certain disorders, additional disease-specific illness and death would occur. One would not expect data from the 13 trials to indicate how often, for example, patients who have acute myocardial infarction secondary to polyarteritis nodosa have various complications of thrombolytic therapy. The usual source in the literature for unusual information is isolated case reports, not the large controlled trials on which predictive instruments are based. Indeed, case reports have described patients who died or developed severe illness when given thrombolytic therapy for acute myocardial infarction due to unrecognized aortic dissection [8-12], acute myocardial infarction due to bacterial endocarditis [13, 14], acute myocardial infarction due to undiagnosed vasculitis [15], acute myocardial infarction with undetected intracardiac thrombi before thrombolysis [16], and acute nonsuppurative and suppurative pericarditis mimicking acute myocardial infarction [8, 9, 17, 18]. The 13 trials did not systematically identify patients with any of the above-mentioned rare conditions, so the predictive tools based on these trials cannot be expected to do so, either. Let the clinician beware!

Developers and users of predictive instruments thus face a quandary when introducing predictive tools into primary care settings. Large randomized, controlled trials are important sources of clinical evidence. Yet the purpose of such trials is to demonstrate overall benefit for enrolled patients, not to determine the specific therapy for every last patient. Predictive instruments, which summarize and make available the distilled experiences of large clinical studies in a patient-specific manner, should be augmented to exclude patients more effectively than the entry criteria for large controlled trials do. Careful analyses should be performed on data from sub-populations of patients who meet study entry criteria but are otherwise not well represented in large trials and who would be harmed if subjected to the targeted interventions. The results of such analyses should be folded back into the predictive instruments. For example, extending Selker and colleagues' predictive tool to exclude patients with acute pericarditis would be worthwhile. Studies suggest that the ST-segment axis may discriminate between patients with acute myocardial infarction and acute pericarditis [19]; if the electrographic criteria used by the predictive tool calculated the ST-segment axis, patients with pericarditis would be less likely to appear to have false-positive entry criteria. Similarly, patients with acute myocardial infarction due to aortic dissection might share characteristics that could be easily identified through review of case reports. Incorporating such information into the predictive tool might prevent the patients from having inappropriate true-positive entry criteria. Ultimately, some exclusions (such as those for vasculitis) will probably have to rely on ad hoc, sound clinical judgment exercised by alert and experienced care givers. To address such circumstances, developers should ensure that when predictive tools print recommendations, a disclaimer listing dangerous situations not considered or covered by the system is also printed [20].

Evidence-based predictive tools will provide future clinicians with many potential benefits. But the proof of the pudding is in the eating. New clinical trials should investigate the best methods for integrating predictive tools into busy clinical practices. Selker and colleagues have taken a good approach to this by suggesting that their predictive instrument should become part of standard electrocardiography analysis devices. If a different handheld device or a different computer application had to be used for each new predictive tool, the number of tools that could be adopted effectively would be extremely limited. It will be important to study which display outputs from predictive tools (for example, relative risks, absolute risks, or odds ratios) maximally influence clinicians' behaviors. Finally, controlled trials must be done to document the benefit afforded by each predictive tool in actual clinical practice.

The purpose of this commentary has been to raise a cautionary note in the minds of clinicians as they adopt otherwise valuable predictive tools. In that regard, Bacon related a pertinent anecdote [7]:

“It was a good answer that was made by one who, when they showed him hanging in a temple a picture of those who had paid their vows as having escaped ship-wreck, and would have him say whether he did not now acknowledge the power of the gods—‘Aye’, asked he again, ‘but where are they painted that were drowned after their vows?’”

• Copyright ©2004 by the American College of Physicians