We appreciate the thoughtful comments of Dr. Evans regarding the potential for evaluation bias. Indeed, the predicted probabilities of significant, severe, and left main disease for patients who had cardiac catheterization were 0.68, 0.28, and 0.06, respectively, compared with 0.47, 0.18, and 0.04, respectively, in patients not referred for cardiac catheterization. As expected, patients unlikely to have these outcomes were not referred as often for diagnostic cardiac catheterization.

Dr. Evans requested that we recalculate the sensitivity and specificity using the Bayesian approach to correct for the verification bias [1, 2]. Figure 1 shows the receiver operating curves for the assessment as presented in our article [3] with the recalculated receiver operating curve correcting for the "verification bias". The two curves are virtually identical. The receiver operating curves are calculated by varying the threshold above which the prediction of significant disease is considered to be a "positive" test result and below which the prediction of significant disease is considered to be a "negative" test result. The sensitivities and specificities vary when calculated for a particular cut point in correcting for the verification bias. Table 1 shows three examples in which thresholds for the likelihood of significant coronary disease are 25%, 50%, and 75%, respectively. At 50%, the sensitivity present in the study patients referred to catheterization was 90% and the specificity was 67%. When corrected for the verification bias, the sensitivity is 79%, and the specificity is 83%. Although the sensitivity and specificity vary in this example, the result is to move to a different place on the receiver operating curve.

View larger version (14K): [in this window] [in a new window]   Figure 1. Unadjusted and adjusted receiver operating curves for coronary artery disease assessment. ROC = receiver operating curve. Solid line = unadjusted; dotted line = adjusted to total population.

We share Dr. Evans' concern about potential sources for bias. Clinicians wishing to apply probability models need to be concerned about three separate problems [4]. The first is the method for generating estimates. The second is the overall quality of the probability prediction. This can be thought of in terms of reliability (how close a given prediction is to the actual value), discrimination (the ability to separate patients with and without the outcome of interest), and precision (the "random noise" or variation in the estimate). The third is generalizability (the likelihood that the results apply to an individual clinician's patient). Even if we had catheterized every patient attending the outpatient Cardiac Diagnostic Unit our patients differ from patients seen in a primary care practice. Consequently, some corrections would still be needed to use these estimates in general practice.

To overcome these problems, we have validated the models in various populations cared for in different referral and evaluation practices, such as Trent, England [5]. Such evaluations are a more rigorous test of the generalizability of model results.