Previous retrospective studies of the accuracy of claims data for the diagnosis of acute myocardial infarction found that clinical criteria were met in 43% to 80% of patients discharged with that diagnosis [3-6]. The Duke study extends these findings because it did not use retrospective chart review to validate the diagnosis but compared claims data with contemporaneously collected clinical data. In addition, the investigators were not constrained in identifying patients on the basis of claims data alone.

Their study involved 12 937 consecutive patients discharged between July 1985 and May 1990 with a procedure code for coronary arteriography. Lacking a third comparison group, they used the cardiology data set as the criterion or "gold" standard because it was likely to be more accurate. Although the assumption probably is correct, purists might give the claims data the benefit of the doubt by assigning accuracy whenever a diagnosis appears in either data set. Recalculating their results using the latter assumption Table 1 does not substantively change their findings. The strongest agreement between the two data sets was in the diagnosis of diabetes: This diagnosis was missing in 15% of the cases in the claims data set and in 9% of the cases in the cardiology data set. For acute myocardial infarction, the discrepancy was 23% and 7%, respectively. The discrepancy for important prognostic variables for ischemic heart disease was even greater. The diagnosis of congestive heart failure was missing in 52% of the cases in the claims data set but in only 19% of the cases in the cardiology data set. For the diagnosis of mitral insufficiency, the discrepancy was 49% and 12%, respectively; for the diagnosis of unstable angina, the discrepancy was 69% and 3%. Overall, more than half the patients with prognostically important conditions were not identified by the claims data set.

Although discrepancies in the coding and documentation of billing data are not new, they were of little concern to most physicians until they began to affect reimbursement and judgments about quality of care. In 1982, Maryland's Health Care Cost Review Commission uniform data sets for hospitals were first used to see if billed diagnoses justified permanent pacemaker insertion. Ralph Nader's Public Citizen Health Research Group concluded that at least 817 of 2222 insertions were unjustified [7]. A re-review of 610 of these questionable 817 placements showed, in 95% of cases, failure to code justifying diagnoses or erroneous procedure coding [8]. A follow-up to a companion report assigning surgical mortality rates to Maryland hospitals [9, 10] also showed many salient coding errors, including instances in which patients were inaccurately reported as having been discharged dead. Since then, the incentives to pay attention to coding have increased, and, as Jollis and colleagues show, the claims data sets have improved. Between 1985 and 1990, the likelihood of identifying important clinical conditions increased from 33% to 46%—good but not great. Much of the improvement was for patients older than 64 years, probably because of Medicare requirements.

The temptation to use databases for research because they are large and include a defined population such as the elderly is understandable [11, 12]. Dissemination of increasingly sophisticated, user-friendly computer hardware and software now permits the manipulation of gigantic databases, leading many to say that we are in the midst of an "information explosion". However, it is more appropriately called a "data explosion" because databases, no matter how large or how available, are not informative unless they are accurate, complete, and can answer important questions. Otherwise, they are distracting and potentially misleading.

More subtle are the undue effects on young investigators, who, because of the pressure to publish, often seek out established databases to mine. The approach seems to be: identify the data first, then proceed to the question. There is even software that promises to find the buried treasure in a database. This approach is the reverse of good science and at its worst can lead to data dredging and misleading posthoc analyses, not because the questions are unimportant but because the database is not up to the task. Indeed, complex statistical methods may lend false credibility to predictive models constructed from under-representative data or unvalidated adjustments for key missing variables.

The challenge, then, is to use large data sets appropriately. The Medicare data set, for example, is useful for analyzing trends in efficiency and charges using such process measures as paid claims, length of stay, and resource use. As for outcomes, when linked to the Social Security files, the Medicare data set can help to identify trends in mortality and to generate explanatory hypotheses [13]. However, to refute or confirm these hypotheses requires specific prospectively collected clinical information that is not readily available.

One major reason people turn to such databases is that their availability makes using them comparatively inexpensive [11, 12], whereas conducting the studies to gather primary data on how patients are managed is costly. In 1977, Congressman Henry Waxman introduced a bill (HR 4869) to create a new institute at the National Institutes of Health (NIH) devoted to studies of cost-effectiveness and technology assessment [14]. He believed that placement within NIH would enhance the prestige of these neglected areas among clinical researchers and at the same time ensure its adequate funding because of the NIH's political standing in Congress. The NIH opposed the bill largely because such applied research was so potentially costly that it might overshadow their basic research mission; instead, NIH created an Office of Medical Applications and Research to convene consensus conferences on important questions but not to produce new data [15]. Simultaneously, Senator Edward Kennedy successfully introduced legislation that passed in 1978 as Public Law 95-623 creating a National Center for Health Care Technology, but the Center survived only 3 years because of political and institutional problems [16]. Since then, the Agency for Health Care Policy Research, established in 1989 [17], has funded studies to determine outcomes in such diverse conditions as prostate cancer, back pain, biliary tract disease, cataracts, and hip fracture. However, allocation for research and development to assure quality outcomes and cost-effective care is probably less than $150 million, or about 0.02% of what is spent on health care. Any meaningful health care reform must substantially increase this percentage.

Outcome studies will require that physicians enter key diagnostic and prognostic data in an accurate and timely manner. This is more difficult than it sounds, not just because of the enormous logistical problems or the need for positive incentives to assure completeness and to inhibit overcoding, but also because of the lack of physician uniformity in naming patient conditions. For example, "unstable angina," an important prognostic indicator, is variably used by both cardiologists and noncardiologists. One doctor's "bronchitis" may be another's "bronchopneumonia". For the most part, these discrepancies reflect differences in physician training and temperament. All of us can recall the range in the admitting thresholds of residents, from low ("sieves") to extremely high ("iron-gates"). Given that, in Osler's words [18], "medicine is a science of uncertainty and an art of probability," it is not surprising that different thought patterns may lead to different diagnostic and treatment strategies. As Chassin [19] has suggested, much of the demonstrated small area variation in medical care may reflect the enthusiasm of local clinician opinion makers, abetted by overly optimistic reporting of research results and financial incentives or patient and family pressures. Such enthusiasm can also be seen as a physician response to Feinstein's "chagrin factor" by using all available resources to minimize regret [20].

Thus, the mandate to link databases [21, 22] and to collect data that are more accurate and complete [23] should be seen not as a means of necessarily yielding "the answer" but of ushering in a much-needed explosion in research on how, why, and to what end physicians do what they do. Indeed, in my experience, good physicians want to know not just "How am I doing?" but "How can I do better?" Unfortunately, clinicians have not been sufficiently involved in framing the important questions and in ensuring that the necessary data are collected. This has generated legitimate fears that extant data sets can be misused by persons who are not themselves accountable to patients to make definitive and too often pejorative judgments rather than to generate hypotheses and to educate physicians and patients. They are right to be concerned. Benefits managers and insurers are now faced with a proliferation of physician profiling programs that rely on ICD-9-CM- disease coding to aggregate patients and to adjust for severity of illness using techniques that have not been widely validated. Researchers involved in such efforts need to be careful, then, not just to use the right data sets but to specify what their studies really show. Although investigators may state the limitations of their data in the discussion section of their papers, such qualifiers do not usually appear in the television and press accounts. In short, as we enter the outcome study era [24], health care policymakers and researchers need to avoid, to paraphrase a 1980 country song, "Looking for answers in all the wrong places".