Evidence-Based Medicine and Large Databases

The challenge is to define the meaning of evidence-based medicine. To the extent that evidence based means that clinical encounters should be supported by scientific conclusions based on data as much as possible, the rationale for evidence-based medicine is self-evident. However, disagreements about the specifics are abundant. In particular, the clinical and health policy communities lack consensus about what types of evidence are relevant to decision making, how to properly evaluate and interpret various bits of evidence, and how to translate evidence into plans of action (such as recommendations and guidelines).

In this context, the large database has emerged as an attractive but controversial source of information. In typical applications, the term large database is synonymous with databases from administrative files (such as Medicare claims), in which the original purpose of data collection was not evaluation of clinical issues. The appeal of these data is that they are readily available and relatively easy to use. Persons who advocate the use of these data anticipate that their value will be revealed through extensive statistical analyses (for example, regression modeling) that can lead to inferences about the relative efficacy, cost, and patterns of use of various clinical interventions [2]. The approach is seductively simple. Yet administrative data are subject to various and well-catalogued sources of bias [3], and their use as a source of information for clinical practice is viewed with legitimate concern.

Given these problems with administrative data, it would be productive to broaden the meaning of large databases to include any database that contains many patients, regardless of the underlying study design. This broader definition includes Medicare claims files, large cohort studies, and meta-analyses of randomized trials that include many patients in aggregate. This definition is more consistent with the goals of evidence-based medicine. However, adaptation of the definition stretches the use of analytic tools typically encountered in database analysis. Rather than analyzing a file with a simple structure (one record for each patient) from a single source, researchers would be faced with various information sources that could have different variables and structures and could have been obtained from different types of study design.

After our view of what data should be considered as evidence has been expanded, the next challenging step (addressed in this article) is to identify a means by which the various data can be effectively used. Our response to this challenge is to develop a comprehensive decision model that combines information from disparate sources into a single structure. Such an analytic approach to decision making invites criticism from investigators who believe that not all decision models are created equal and that some decision models resort to oversimplification and unrealistic assumptions. This criticism is fair because an information tool with the power of decision modeling carries substantial dangers. Certainly, model building should always be done in a systematic fashion, with special consideration given to such issues as consistency with underlying clinical data, documentation, reproducibility, and model validation.

A Comprehensive Approach Applied to Decisions in Stroke Prevention

Every year, approximately 500 000 persons in the United States have a stroke and 150 000 persons die as a result. Stroke is the third leading cause of death in the United States and is the leading cause of serious disability. At an annual cost of at least $30 billion to $40 billion, stroke is expensive [4, 5]. Moreover, stroke is a widely feared event: more than 40% of respondents to a recent survey rated a hypothetical major stroke to be a worse outcome than death [6]. Because strokes predominately affect elderly persons, the clinical and public health importance of stroke can be expected to grow as the number of elderly persons continues to increase.

One of the goals of the Patient Outcomes Research Team for the Secondary and Tertiary Prevention of Stroke (Stroke PORT) is to evaluate the cost-effectiveness of surgical and medical interventions that prevent stroke. We have organized tasks related to this goal around a comprehensive model of stroke development and outcome: the Stroke Prevention Policy Model (SPPM) [7-11] (Figure 1). This model uses various data as inputs, including epidemiologic studies, randomized clinical trials, administrative data, and patient interviews. Many of these inputs represent large databases. The SPPM outputs include estimates of the health and economic outcomes of alternate stroke prevention practices. By linking clinical evidence (in particular, from population-based studies, claims data, and large randomized trials) to the health outcomes of individual persons, the SPPM is intended to improve the health care community's understanding of the available prevention options, thereby focusing on and improving the ability of clinicians, patients, and policy makers to make informed decisions about stroke prevention.

View larger version (23K): [in this window] [in a new window]   Figure 1. Structure of the Stroke Prevention Policy Model (SPPM). The link between inputs (clinical trials, epidemiologic studies, administrative data, surveys of patient preferences, and expert opinion) and outputs (survival, quality-adjusted survival, and cost) is shown.

The focus of this article is the role of large databases (as broadly defined) in the development and operation of a comprehensive decision-modeling effort-the SPPM. Our article consists of three sections. First, we describe the SPPM in terms of its basic structure and function. Because this article focuses on the application of large databases in the context of the SPPM effort rather than on the SPPM itself, this description is illustrative rather than comprehensive (more comprehensive descriptions are published elsewhere [8, 9]). Next, recognizing that the use of modeling continues to be a source of controversy within the medical community, we discuss the philosophical underpinnings of the SPPM. Finally, we consider how some of the lessons learned from the SPPM might be applied to simulation-based decision modeling for similarly complex issues.

Description of the Stroke Prevention Policy Model

Component Models

The overall SPPM incorporates component models for the natural history of cerebrovascular disease (natural history model), implications of prevention strategies (intervention model), patient preferences (utility model), and costs (cost model).

The natural history model (the backbone of the SPPM) describes the development of cerebrovascular disease in the absence of specific preventive interventions. The intervention model describes how these interventions modify the natural history of stroke. By linking the natural history model (as possibly modified by information about interventions) with the patient preference and cost models, the SPPM can be used to evaluate the cost-effectiveness of various intervention strategies.

Natural History Model

Structure

The SPPM is a semi-Markov simulation model used to support decision and cost-effectiveness analyses. Its fundamental elements consist of health states and events, whereby events represent transitions between health states. Because of its technical complexity, the SPPM operates by simulating the natural history of a large cohort of patients. The health-related history of an individual patient is random (that is, it is based on the simulation's generation of random numbers) yet follows underlying transition probabilities. As the size of the simulated cohort increases, the relative effects of chance being caused by the simulation are reduced and the natural history of patients can be described with any desired degree of precision.

To compare an intervention with standard practice, we simulate the experience of a large cohort of patients using the natural history model, modify the parameters of the natural history using the intervention model, and then repeat the simulation. We use a similar strategy to compare two interventions.

Events

The SPPM takes into account the following clinical events: transient ischemic attack, ischemic stroke, hemorrhagic stroke, myocardial infarction, complications of treatment for disorders other than stroke or myocardial infarction (for example, gastrointestinal bleeding as a complication of anticoagulation), and death (which can be the result of stroke, myocardial infarction, or other causes). Patients who survive at least 30 days after a stroke are randomly assigned a stroke-related disability level defined by the Rankin score [12]. Patients who die of a stroke within 30 days are assigned a Rankin score of 5 (indicating the highest possible stroke-related disability).

The health states that represent fundamental elements of the model can be defined from these events in various ways. For example, a patient who has had both a transient ischemic attack and an ischemic stroke might be defined as being in the "transient ischemic attack and ischemic stroke state." In contrast, the same patient might be defined as being in the "transient ischemic attack state" and then undergoing a transition to the "ischemic stroke state." For convenience, we adopt the latter convention and focus on the following codes for the health states: ASY (asymptomatic), TIA (transient ischemic attack), IS (ischemic stroke), HS (hemorrhagic stroke), MI (myocardial infarction), and DTH (death). As a notational convention, although the SPPM records complication-related events, we do not consider complications (COMP events) to be a fundamental health state because we assume that treatment-related complications affect utility and cost but not probabilities of transition between the other health states.

Transition Probabilities

Figure 2 shows the rates of time-dependent events (transition probabilities) that describe the likelihood of transitions among the various health states in the SPPM. For simplicity of presentation, the graphs in Figure 2 are structured as cumulative probabilities of transition for patients who are 70 years of age at the beginning of follow-up and have representative values of the other covariates. Of note, the graphs in Figure 2 present the probability of sentinel health events that are conditional on the next event being the transition in question. For example, the graph that shows transitions from IS to IS (that is, repeated ischemic stroke) describes the likelihood that IS will occur again within a certain time after the initial IS, assuming that patients who have had MIs (or other SPPM-relevant event) in the interim have been censored (eliminated from follow-up) at the time of the MI (or other SPPM-relevant event).

View larger version (29K): [in this window] [in a new window]   Figure 2. Matrix of cumulative probability distributions. Transitions to the same health state (for example, IS to IS) indicate a repeated event. Patients are eligible for transitions between any of these health states; however, patients cannot return to the ASY state after experiencing an event relevant (by definition) to the Stroke Prevention Policy Model. In addition, patients cannot make a transition from TIA to TIA (based on the clinical determination that TIA is a transient event). Patients are eligible for all transitions to and from HS. However, because of data limitations (that is, relatively few HS events in the data sources-only 15% of strokes are HS) and because the focus of the SPPM is on IS rather than on HS, the occurrence of HS is treated as statistically independent of other transitions in the SPPM. In other words, all transitions to HS events use the cumulative probability distribution of ASY to TIA. All transitions from HS use the cumulative probability distribution based on the preceding state. ASY = asymptomatic; DTH = death (from any cause); HS = hemorrhagic stroke; IS = ischemic stroke; MI = myocardial infarction; and TIA = transient ischemic attack.

The model recognizes that transition probabilities depend not only on the current health state and time since the previous event but also on various demographic and clinical variables (covariates) unique to the patient. Covariates considered by the SPPM include age, sex, race, tobacco smoking, systolic blood pressure, hypertension, cervical bruit, non-invasive or angiographic evidence of occlusive carotid artery disease, diabetes mellitus, coronary artery disease, atrial fibrillation, congestive heart failure, peripheral vascular disease, and valvular heart disease. Covariates also include the following sentinel events: previous TIA, previous MI, previous HS, and previous IS. The selection of covariates was based on published evidence and expert opinion with regard to features that would influence natural history, preference for outcomes, cost, response to tests or treatments, or any combination of these.

Although most patients have events during the short to medium term, the SPPM requires that transition functions be extrapolated over longer periods because each patient is followed until death. In the absence of comprehensive long-term data, we assumed that if no events occurred within 10 years, the hazard function reverts to that of the asymptomatic population. The hazard function in the medium term (between 5 and 10 years) is an average of long- and short-term hazard functions.

Transition probabilities that incorporate the effect of covariates were primarily estimated by applying the Cox proportional-hazards modeling technique to data from the Framingham Study [13]. Because previous reports from the Framingham Study did not include results in the format used by the SPPM (that is, censoring other SPPM-relevant events), we submitted an analysis request asking the Framingham investigators to reanalyze their data in this manner and to provide the SPPM with the resulting model estimates. The mechanism of an analysis request allowed Framingham investigators to maintain the confidentiality of their database while facilitating the inclusion of data from one of the largest and most meticulously performed population-based cohort studies into the SPPM. The Cox proportional-hazards models estimated by the Framingham investigators included a nonparametric estimate of the shape of the transition function (as shown in Figure 2) and regression coefficients that reflect the contribution of the various covariates [13]. Other large databases used to validate the SPPM transition probabilities were obtained from the Olmsted County Epidemiology Project (again, after a request to reanalyze data censoring other SPPM-relevant events) and NASCET (North American Symptomatic Carotid Endarterectomy Trial Collaborators), which served as a control group [14].

Intervention Model

To determine the implications of various prevention strategies, we formed intervention models by modifying the transition probabilities in the natural history model. This modification is typically accomplished by using risk ratios and complication rates that quantify the efficacy of the intervention in question. For example, suppose that the probability of a symptomatic patient having an IS within the next year is 5%. If the risk ratio for anticoagulation is 0.5, then the corresponding probability of IS within the anticoagulation model is 2.5% (that is, 5% x 0.5). Interventions can lead to complications other than MI and HS. For example, anticoagulation can lead to substantial gastrointestinal bleeding. These effects were accommodated through the COMP event in the SPPM. The occurrence of a complication was assumed to have no effect on transition probabilities for other sentinel events.

Parameters of the intervention model were obtained, when possible, from the meta-analysis of randomized trials (for example, trials that assessed anticoagulation management in patients with atrial fibrillation and carotid endarterectomy in patients with symptoms of cerebrovascular disease [15, 16]). Meta-analysis serves to increase the effective sample size available for analysis. For example, one of the strengths of meta-analysis is that it can, in effect, make a large database from a set of small- to moderate-sized studies. We chose not to use administrative data to estimate intervention parameters and preferred instead to rely on expert judgment to complete the relatively few parameters that could not be obtained from the literature. The decision to use large administrative databases to estimate utilization and cost but not treatment efficacy is an example of the underlying principle behind SPPM: to use each data source for its best purpose. This principle recognizes both the inherent biases involved with nonrandomized designs to measure efficacy and the strength of using administrative databases for describing utilization and costs.

In keeping with our desire to make the model a general-purpose tool, the SPPM was constructed to permit the evaluation of various stroke preventive interventions. These interventions include diagnostic testing (such as noninvasive studies of the carotid arteries and angiography), medical treatments (such as use of antiplatelet and antithrombotic agents), and surgical interventions (such as carotid endarterectomy and carotid stenting). Our model also accommodates examination of arbitrarily complex sequences of interventions that depend on time and patient characteristics. For example, with the SPPM, we can examine the use of serial noninvasive testing of the carotid artery at 5-year intervals for asymptomatic persons with moderate stenosis on preliminary examination.

One of the advantages of using a natural history model based on simulations is its ability to accommodate many potential interventions with minimal reprogramming. The SPPM users are not limited to the initial set of interventions that we considered (for example, medical management through anti-coagulation and aspirin therapy and surgical intervention through carotid endarterectomy, including the pattern of invasive and noninvasive tests that preceded surgery). In fact, the approach of using special parameters to modify the results of the natural history model can be applied elsewhere in the SPPM (for example, studying the effect of rehabilitation on patterns of utilization and outcomes after a stroke).

Utility Model

One of the principal goals of the SPPM is to project health outcomes that are relevant to policy formation. Consistent with the recommendations of recent expert panels on cost-effectiveness methodology, the SPPM has been constructed to facilitate an estimation of cumulative quality-adjusted life years (QALYs) [17] that are associated with each policy alternative.

To obtain quality-adjusted life expectancy, we assign a quality adjustment factor (that is, a utility weight) to each health state in the SPPM. The counting routine then uses these factors to transform total life expectancy into quality-adjusted life expectancy. The QALYs were discounted at the real riskless rate of 3% [17].

To estimate the weights of utility for each stroke state, we interviewed (using the time tradeoff approach) more than 1200 persons at elevated risk for new or repeated for stroke to obtain their relative preferences for stroke-related outcomes [6]. To our knowledge, this is the largest such survey of the preferences of patients at risk for stroke. Utility weights for myocardial infarction and other health states were obtained from the literature.

Cost Model

To determine the cost of each stroke prevention strategy, we attached a cost to each patient history on the basis of both the health states involved and the duration of each health state. Costs include total direct medical and nonmedical resource costs discounted at the real riskless rate of 3% [18].

Direct medical costs have been estimated from various sources of claims data, including the Medicare-based National Claims History Files, which focus largely on acute care and rehabilitation services for elderly persons; United HealthCare, which focuses primarily on acute care for nonelderly persons; the Rochester Epidemiologic Project, which mainly estimates nursing home and other long-term care costs incurred by stroke patients in Olmsted County, Minnesota; and the Academic Medical Center Consortium, which obtains detailed estimates of the cost of caring for stroke at major teaching hospitals. Direct nonmedical costs were calculated from the published literature on caregiver burden, home modification, and other expenditures not channeled through the health care system.

Why Should the Stroke Prevention Policy Model Be Used?

Why should the medical profession accept quantitative modeling procedures that incorporate empirical evidence, such as the SPPM, rather than continue to rely primarily on the intuition of experienced physicians? In this section, we will argue that the rationale for such an approach is that it provides a general framework for supporting (but not necessarily prescribing) clinical decision making. As a result of the formal structure of the SPPM, more uniform clinical policies (typically in the form of guidelines) are easier to develop. These policies may ultimately serve the needs of patients, physicians, and society as a whole. If guidelines are developed and accepted, then patients can expect to receive treatment on the basis of more consistent procedures, thereby enhancing the probability that patients with similar conditions will receive similar treatment (an ideal of distributive justice). In addition, patients can take comfort in that their preferences have been accommodated by the SPPM's utility model in an effort to show respect for the ethical principle of personal autonomy. Likewise, physicians can be satisfied that their clinical decisions are based on the best evidence currently available, thereby ensuring that patients are receiving treatment according to the ethical principle of beneficence. Finally, society at large benefits because the use of these clinical policies enables comparisons among providers, thereby helping to achieve the goal of cost-effectiveness in the delivery of health care services. Because the structure of the SPPM facilitates the development of policies that take into account these diverse principles, we argue that acceptance of a decision modeling approach is ethically appropriate [19, 20].

Our model represents an extension of other efforts to establish comprehensive decision models [21-23]. We contend that this approach is, in turn, a logical extension of what has been called evidence-based medicine [24]. Nevertheless, both philosophical and practical barriers to the approach do exist. Within the medical profession itself, there has been a long-standing philosophical suspicion of these types of models. Although a few traditional statisticians may question the Bayesian perspective on which the SPPM is predicated [25], the more common criticism is that such a quantitative approach based on numerous data sources fundamentally violates the art of medicine as practiced by individual physicians. In criticizing these complex quantitative models that claim to represent the best evidence known, opponents emphasize the inherently inchoate and amorphous character of medical judgment. They argue that modeling is no substitute for personal experience and intuition [26].

The critique of decision modeling and evidence-based medicine can also be framed in ethical terms. Critics emphasize that concrete ethical norms govern the professional behavior of physicians, particularly the personal nature of the physician-patient encounter, which is viewed as central to the art of medicine. As previously mentioned, the use of tacit knowledge in medicine has traditionally been geared toward individualized patient care [1]. Therefore, the adoption of explicit and uniform guidelines that is implied by accepting such tools as the SPPM might undermine the inherently private character of the traditional physician-patient encounter. Taken in its extreme form, this argument suggests that any statistically derived inferences are an anathema to medicine as a private art.

Although the claim that the development of uniform guidelines may alter the nature of the clinical encounter has some merit, we respond that such alteration is not necessarily bad. The idea that the art of medicine is a purely private activity conducted solely between a specific physician and patient is hardly realistic today. A host of other players, ranging from hospitals to insurance companies to government, now actively participates in medical encounters. Because contemporary medicine has become a vast industry [27] and medical decision making has emerged as an issue of public policy, the need is greater than ever to impose some degree of uniformity on medicine through such models as the SPPM rather than continuing to rely principally on the expertise of individual health care providers.

As a historical aside, these philosophical debates over the role of quantification in medicine illustrate the problem of old wine in new bottles. Debates about the relative merits of practicing medicine using personal medical knowledge in contrast to quantitative decision-making procedures dates back at least two centuries [28]. Eighteenth-century mathematicians, such as Pierre Simon Laplace, argued that probability theory was "good sense reduced to a calculus" [29] and eerily foreshadowed the views of contemporary proponents of medical decision analysis, such as Howard Raiffa, who argued the following:

"To me decision analysis is just the systematic articulation of common sense: Any decent doctor reflects on alternatives, is aware of uncertainties, modifies judgments on the basis of accumulated evidence, balances risks of various kinds, considers the potential consequences of his or her diagnoses and treatments, and synthesizes all of this in making a reasoned decision that he or she decrees right for the patient. All that decision analysis is asking the doctor to do is to do this a lot more systematically and in such a way that others can see what is going on and can contribute to the decision process [30]."

The Lessons To Be Learned

The debate about whether scientific evidence should have a central role in clinical decision making is a quaint historical footnote from an era in which the absence of evidence made such discussions largely academic. Increasing amounts of data and general agreement among the health care community support the concept of evidence-based medicine. Today, the pressing concern is how scientific evidence should be interpreted and subsequently woven into the practical process of providing health care. In this article, we have argued that strong ethical reasons support the pursuit of comprehensive approaches to information synthesis in medicine: Decisions need to be made, and preference should be given to methods that link evidence to those decisions. The approaches represented by the SPPM provide a comprehensive solution to the practical problem of information synthesis and therefore deserve serious attention from the medical community.

Stroke prevention is particularly amenable to modeling approaches because complex medical issues associated with stroke can hinder physicians who rely solely on personal experience to make optimal choices and because of the wealth of excellent information available to be synthesized. One reason that issues in stroke prevention are complex is because of the tradeoffs involved in the decision-making process, for example, in comparing the inconvenience and increasing likelihood of bleeding associated with anticoagulation management with the decreased likelihood of ischemic stroke and in comparing the short-term risks, disutility, and cost of carotid artery endarterectomy with its long-term benefits. Complexity is also a reality in that outcomes, both clinical events and costs, can occur far into the future.

The wealth of information pertinent to stroke prevention includes large randomized trials [15, 16], Medicare claims files and other large sources of administrative data, and large population-based cohort studies. The structure of the SPPM facilitates using each data source to full advantage (for example, we used administrative files to describe patterns of utilization and cost but not to estimate efficacy). This model allows the results of each data source to be summarized by a few simple parameters, thereby effectively allowing data from widely different designs and subpopulations to be combined in a single all-inclusive framework. The Table 1 summarizes the data sources currently used as inputs into the SPPM.

The SPPM model represents a methodologic advance in decision modeling on stroke because of its realistic parameters and ability to generalize applications [7-11]. Whereas all mathematical models use simplifying assumptions, the SPPM is designed to closely detail the underlying complexities associated with the natural history of stroke and preventive strategies. On the other hand, because the SPPM design is based on the natural history of stroke, it can be used to assess the implications of various stroke prevention strategies without the need to develop separate models for each new analysis.

We recognize that all models are not created equal and that using inaccurate, biased, or unvalidated decision models can lead to misleading conclusions. Accordingly, we have emphasized that the SPPM combines a coherent structure with the best available data on stroke prevention. In this article, we do not describe the meticulous attention to methodologic detail that went into the design of the SPPM, the model's ability to quantify the imprecision of its estimates and thereby avoid the illusion of overly precise conclusions [10], or the numerous steps involved in validating the model [11]. However, we do emphasize the necessity for model validation by using both internal and external sources before policy conclusions can be seriously considered.

We do not propose that the SPPM (or any model) become a substitute for human judgment, either in the formation of clinical policy or in individual decision making. Instead, we argue that such models are a tool to facilitate thoughtful and satisfying decision making [31]. They provide a potentially vivid picture of the dynamics of complex decisions and help define the limits of reasonable speculation that is based on available data and expert opinion. As a result of a comprehensive process of development and validation, we argue that component parts of the SPPM fit together well enough to enable the model to address the real concerns of the clinical decision makers it was designed to serve. Therefore, the model has achieved the requisite amount of conceptual coherence necessary to be adopted as a heuristic tool to complement the insights provided by more conventional forms of analysis. In other words, the SPPM is "sufficient in form and content to resolve the issues" for the decision maker [32].

Any new argument in favor of such approaches as the SPPM is unlikely to develop. Therefore, we suggest that the principal means for convincing a wider audience of the utility of linking large databases by decision modeling is not to engage in more philosophical theorizing but rather to generate concrete practical results that will actually aid clinical decision makers. Development of these models still faces major hurdles. Model development is a tedious process that has numerous steps and frequently requires resolution of such issues as the proprietary rights and confidential nature of various databases. In addition, the limited availability of software and the desire to avoid oversimplification often necessitate the development of new special-purpose computer code. Finally, performing extensive validation is time-consuming and requires the use of external data sets.

Despite these difficulties, we believe that the benefits (for example, in terms of better stroke prevention) of developing such models still outweigh the costs of continuing to rely on more conventional methods. Modeling provides a common framework for enlarging the pool of data and offers a guide to future research and modes of analysis. In this way, a well-conceived, comprehensive model can improve clinical decision making today and in the future.

Dr. Samsa: Division of Biometry, Department of Community and Family Medicine, Duke University Medical Center, First Union Tower, Suite 230, 2200 West Main Street, Durham, NC 27705.

Dr. Ancukiewicz: Division of Radiation Oncology, Department of Medicine, Massachusetts General Hospital, Founders Building #516, 55 Fruit Street, Boston, MA 02114.

Dr. Parmigiani: Institute of Statistics and Decision Sciences, Duke University, Box 90251, Durham, NC 27708.

Dr. Wolf: Section of Preventive Medicine and Epidemiology, Evans Memorial Department of Clinical Research and the Department of Medicine, Boston Medical Center, 715 Albany Street, Room B608, Boston, MA 02118.

Dr. D'Agostino: Department of Mathematics, Boston University College of Arts and Sciences, 111 Cummington Street, Boston, MA 02215.

Dr. Lipscomb: Sanford Institute of Public Policy, Duke University, Box 90245, 212 Sanford Institute Building, Durham, NC 27708.