Our purpose was to estimate the sensitivity and specificity of noninvasive carotid artery tests using formal meta-analysis and to determine which tests are the most discriminating in the detection of carotid artery occlusion and severe disease (defined as >70% stenosis) using composite receiver operating characteristic (ROC) analysis and a summary measure of test effectiveness.

Methods

Top Methods Statistical Analysis Results Author & Article Info References

Noninvasive Tests Selected for Study

The six noninvasive tests we selected assess carotid artery stenosis either by detecting the carotid artery, the flow within it, or both (direct tests) or by evaluating flow through collateral vessels (indirect tests). Carotid Doppler ultrasonography assesses the velocity changes in blood flow associated with stenosis in the carotid arteries using either a continuous wave, a single-gated pulsed wave, or directional color modes [7-33]. Real-time B-mode ultrasonographic imaging allows for direct visualization of the carotid artery and calculation of the widths of obstructed and unobstructed arteries and of the lesions themselves; thus, investigators can determine the percentage of stenosis [34, 35]. Duplex ultrasonography combines the direct visualization capabilities of B-mode ultrasonography and the blood-flow velocity measurements of Doppler ultrasonography [26-28, 31, 36-56]. Magnetic resonance angiography is a relatively new technique that directly assesses both carotid and intracranial arterial stenosis by building up images from many thin-layer, two-dimensional slices or from a smaller quantity of three-dimensional volumes [29-31, 55-67].

The tests that indirectly assess the carotid artery include supraorbital Doppler ultrasonography and oculoplethysmography. Supraorbital Doppler ultrasonography was the first noninvasive technology available to evaluate carotid artery stenosis; it indirectly assesses blood flow from collateral branches of the internal carotid artery through the supraorbital vessels. The test is done by placing a directional Doppler probe over a supraorbital artery and observing the flow with and without compression of neighboring arteries [32, 68-71]. Oculoplethysmography indirectly evaluates the patency of the internal carotid artery by graphically recording ocular pulses obtained from corneal cups held in place by mild suction. Concomitant ear-lobe pulses are also recorded for timing comparison [33, 51, 70, 71]. The results of these studies can be translated into estimates of carotid patency.

Identification of Articles

We searched the MEDLINE database for English-language articles about the diagnostic testing of extracranial carotid artery disease that were published between 1977 and 1993. The keywords used were oculoplethysmography, ultrasonography, digital subtraction angiography, cerebral angiography, and magnetic resonance imaging (as they relate to disease involving the cerebral arteries); and vertebral arteries, basilar arteries, and carotid arteries in humans. Additional articles listed in the bibliographies of standard neurology texts and references cited in accepted articles were also included among the articles considered.

All articles passed through a multilevel, systematic review by teams of nurses and physicians (Figure 1). Articles were excluded if 1) results from the test used were not compared with the results of conventional carotid angiography or intra-arterial digital subtraction carotid angiography; 2) the angiographic results were not separated to allow for specific identification of occluded arteries; or 3) the reference standard test results could not be classified into a contingency table according to degree of stenosis. Through the MEDLINE, textbook reference, and bibliography searches, we initially identified 568 articles, 354 of which were initially rejected, either because the noninvasive test was not compared with carotid angiography or because carotid artery occlusion was not reported as a separate category. The remaining 214 articles were then analyzed by one of four physicians. An additional 110 of these articles were eliminated, most because the reported data were not sufficient to permit construction of contingency tables. When an article was included, data about patient demographics, study design, sites of patient enrollment, blinding of reviewers to test outcomes, and specific results were abstracted. The reliability of the first exclusion was assessed by randomly selecting 22 (10%) of the rejected articles and submitting them to another physician for a second review. Agreement about the appropriateness of the exclusions was 100%.

View larger version (31K): [in this window] [in a new window]   Figure 1. Article review flow chart.

The 104 articles initially included by the first physician reviewers were then examined and abstracted by second physician reviewers. Discrepancies between the two reviews were resolved by discussions. As a result of these discussions, 34 additional articles were rejected, 15 because contingency tables could not be reliably constructed from the reported data and 18 because of overlapping populations (in these cases, the most recently reported of the studies or the study with the larger population was included). One article was rejected because the diagnostic test reports were given as a single result that reflected the output of many concurrent tests. Therefore, the results of 70 studies form the basis of this review.

Quality Criteria for Evaluating Study Methods

Criteria used to rate the quality of the study methods were based on previously defined standards [72]. Each article was rated on the methods used by the investigators, the site of patient enrollment, the degree of blinding between interpretation of the noninvasive test and the reference standard, and the disease spectrum of the patients enrolled. In most studies, the site of patient enrollment was the radiology or surgery unit. Methods of patient enrollment were specified or implied in all 70 articles: Forty-one percent used consecutive series and 59% used convenience or retrospective samples. Blinding of the reviewers was rarely mentioned in any study; because of the sequence of enrollment, we speculated that the reviewers interpreting noninvasive tests were blinded to the results of invasive tests in 84% of the studies. The indications for the testing of patients were listed in 53% of the studies and included recent stroke or transient ischemic attack and asymptomatic carotid bruits. In a secondary analysis, we examined the results of the meta-analysis by comparing studies in which the reviewers interpreting the noninvasive tests were blinded to the angiograms with those in which they were not blinded and by comparing studies in which patients were enrolled consecutively with those in which another method of patient enrollment was used.

Statistical Analysis

Top Methods Statistical Analysis Results Author & Article Info References

Results from angiographic and noninvasive tests were classified into as many of the following categories as possible: carotid artery occlusion, 99% to 70% stenosis, 70% to 50% stenosis, and 50% to 0% stenosis. Sensitivity and specificity were the primary measures of test performance; carotid angiography was the reference standard. To facilitate comparisons among noninvasive tests, values for sensitivity and specificity (with CIs) were calculated by classifying all lesions above a given cut-point as "positive." For the purposes of this comparison, occlusion was considered positive [6].

Articles relating to one particular test type were summarized by pooling estimates of sensitivity and specificity [73]. Confidence intervals were derived using exact methods for proportions [74]. A chi-square test for homogeneity was done on the data from each study at the 100%, 70% (or equivalent), and 50% stenosis cut-points. Sensitivity and specificity were estimated using a random-effects or a fixed-effects model according to the homogeneity of the test results [73, 75]: When the pooled estimate was heterogeneous, a random-effects model was used.

Composite ROC curves were constructed; the reference standard of carotid angiography was defined as positive at 70% stenosis. This cut-point was chosen because of the entry criteria for the North American Symptomatic Carotid Endarterectomy Trial (NASCET) [3]. Noninvasive tests were then selected as positive at 50% stenosis, 70% stenosis, or carotid artery occlusion, and true-positive and false-positive rates were calculated because these distinctions provide the most information for clinicians who choose to refer patients for angiography at different thresholds. The tests for homogeneity and methods of meta-analytic combination mentioned earlier were then used to derive the pooled true-positive and false-positive rates for each test type [75, 76]. These were plotted as ROC curves, and the points were connected using a nonparametric assumption [77]. Areas under the ROC curves were derived using a trapezoidal method [77, 78].

We also calculated a new measure for diagnostic tests termed "test effectiveness," which attempts to avert the forced tradeoffs between sensitivity and specificity by fitting a ROC curve through a logistic odds transformation of sensitivity and specificity [75, 79, 80]. The result is a measure that is more normally distributed than either sensitivity or specificity; this single statistic can be thought of as a measure of the discriminatory power of the test. To standardize this measure, we adjusted the standard deviation of the logistic normal distribution curve (3/) to get the following measure:

delta = (radical[3]/) x (ln [Sensitivity/(1 –Sensitivity)] + ln

(Specificity/[1 –Specificity]))

where the test-effectiveness statistic delta is derived with associated CIs using maximum likelihood methods [75, 78]. Test effectiveness is interpreted as the standardized distance between the means of the test results for diseased and nondiseased populations. Because the scale is standardized, tests can be compared in relative and absolute terms. In general, a test with an estimated delta value of 1.0 is not effective for discriminating between patients with and without disease, whereas a test with a value of 3.0 is highly effective [75]. An effectiveness score of 3.0 indicates that the mean scores of diseased and nondiseased patients are separated by three standard deviations.

To illustrate the meaning of the effectiveness score, we simulated one diagnostic test with an effectiveness score of 1.0 and one with a score of 3.0. Figure 2 shows a hypothetical distribution of diagnostic test results in two populations of patients: diseased and nondiseased. The first panel shows the test with a score of 1.0, in which diseased and nondiseased patients have significant overlap (27%), which is caused by imprecision in the diagnostic test. When the effectiveness score for the test is 3.0, the overlap is reduced to 3%, implying that the test is significantly more precise.

View larger version (24K): [in this window] [in a new window]   Figure 2. Illustration of the effectiveness score relative to a hypothetical diagnostic test.

Results

Top Methods Statistical Analysis Results Author & Article Info References

View larger version (27K): [in this window] [in a new window]   Figure 3. Meta-analytic sensitivity and specificity for the noninvasive carotid artery tests by degree of carotid artery stenosis. Angio = angiography.

Because some clinicians refer patients for angiography when a noninvasive test identifies less serious stenoses, we also estimated test operating characteristics for 50% stenosis. At this degree of stenosis, carotid duplex ultrasonography, carotid Doppler ultrasonography, and magnetic resonance angiography showed the following operating characteristics: sensitivities ranging from 0.85 to 0.93, specificities at 0.92, and narrow CIs. Supraorbital Doppler ultrasonography, B-mode ultrasonography, and oculoplethysmography had lower operating characteristics.

The composite receiver operating characteristics for the noninvasive tests are shown in Figure 4. For the reference standard, angiography, 70% stenosis was defined as clinically significant. The four tests that showed a high level of accurate discrimination were carotid duplex ultrasonography, carotid Doppler ultrasonography, magnetic resonance angiography, and supraorbital Doppler ultrasonography (equivalent areas under the ROC curve ranged from 0.92 to 0.94). The other two tests, B-mode ultrasonography and oculoplethysmography, did not do as well; their areas under the ROC curve ranged from 0.81 to 0.82.

View larger version (36K): [in this window] [in a new window]   Figure 4. Composite receiver operating characteristic curves for the noninvasive carotid artery tests. Numbers in parentheses are the areas under the composite receiver operating characteristic curve; carotid angiography is the reference standard, and 70% stenosis is the threshold of significance.

View larger version (29K): [in this window] [in a new window]   Figure 5. Effectiveness scores for the noninvasive carotid artery tests separated by degree of carotid artery stenosis. Angio = angiography.

Limitations, Safety, and Cost

We estimated 1994 Medicare reimbursements for the noninvasive procedures on the basis of schedules for participating physicians. These values represent Medicare reimbursement for technical and professional fees, and the amount often differs by a factor of two to three from what the patient may be billed for the procedure. Medicare reimburses $58 for oculoplethysmography, $58 for supraorbital Doppler ultrasonography, $70 for B-mode ultrasonography, $157 for both carotid duplex and Doppler ultrasonography, and $408 for magnetic resonance angiography.

Conclusions

Carotid duplex ultrasonography, Doppler ultrasonography, and magnetic resonance angiography have high sensitivity and specificity for the detection of clinically important stenosis. All three tests were found to be superior in the identification of stenoses greater than 70% of carotid diameter.

The clinically important question is, How do noninvasive tests fit into an overall strategy for evaluating a potential surgical candidate? In extreme cases, noninvasive testing may have little or no role in patient evaluation. For example, a patient who would never accept a prescribed carotid endarterectomy will not benefit from the noninvasive test result. Alternatively, a physician or patient may be convinced that current symptoms are due to ipsilateral carotid artery stenosis and may therefore be unwilling to accept the remaining uncertainty after a negative noninvasive test result (such as a false negative). In such cases, only a definitive angiogram will suffice. However, most decisions are made on the playing field between these two extreme situations. What do noninvasive tests tell us about patients in this middle ground?

First, noninvasive tests may permit the more effective use of angiography for patients found to have mid-range stenoses. A possible screening strategy would be to refer to angiography all patients above a certain cut-point on noninvasive testing. When the cut-point is set at greater than 70% stenosis, four tests appear to have the best operating characteristics: carotid Doppler ultrasonography, carotid duplex ultrasonography, magnetic resonance angiography, and supraorbital Doppler ultrasonography. The sensitivities of these tests at this level are better than they are for occlusion (0.89 to 0.94), but the specificities are lower (0.85 to 0.92). Most of the benefit of using noninvasive tests to screen for disease accrues to those persons with negative noninvasive test results (because the alternative is to proceed to angiography for all patients). Assuming the 28% underlying prevalence of high-grade carotid stenosis (70%) seen in this study, 66% of all noninvasive tests would be negative. Three percent of the group with negative noninvasive test results would have disease that is potentially responsive to surgery but would be misclassified. The remaining 97% of those with negative noninvasive tests would be spared the potential morbidity and cost of angiography.

Second, noninvasive tests do not tell us enough about occlusion to guide decision making. Chronic occlusion of the carotid artery is not suitable for surgery. The best noninvasive tests—carotid Doppler ultrasonography, carotid duplex ultrasonography, and magnetic resonance angiography—all do well at identifying patients with occluded carotid arteries. The sensitivities of these tests for detecting occlusion are high (0.82 to 0.86), as are the specificities (0.98), and their summary effectiveness measure is the best for any category of carotid stenosis (>3.0). Nevertheless, if the prevalence of occlusion in the population is similar to that of the patients in the studies reviewed here (10%), then failing to refer patients who appear to be occluded on a noninvasive test would result in misclassifying 9% of patients who have some degree of patency as having an occlusion. Patients with presumed occlusion on a noninvasive test may still be appropriate candidates for angiography because they may actually have a tight stenosis.

Lastly, the performance of noninvasive tests for screening purposes is not improved by lowering the cut-point for referral to angiography (Figure 5). This strategy will provide a higher sensitivity, but specificity will decrease rapidly. The net result is few additional true positives and many false positives unnecessarily referred for angiography.

Noninvasive tests cannot substitute for angiography as the sole preoperative test for carotid endarterectomy. Noninvasive tests have less-than-perfect performance at classifying diseased and nondiseased patients. The consequences of misclassification are high; endarterectomy is not an entirely benign procedure. Finally, evidence suggests that angiography provides additional information about surgical risk of endarterectomy [81-83]. For example, the presence of intracranial stenosis may double the operative risk of endarterectomy and therefore reverse the clinician's and patient's decision for surgery [82].

Our study had several limitations, some that are inherent to meta-analyses and some that are unique to this subject. First, we have included information only from published series. The centers and investigators who chose to report results of the various noninvasive tests may be remarkable in their ability to do accurate studies. This "publication bias" generally results in overestimating the true sensitivity, specificity, and effectiveness measures of the tests. Second, we were unable to adjust the sensitivity and specificity of any of the tests for "verification bias" [84], which occurs when patients with negative or normal noninvasive test results fail to receive the reference standard, angiography. We were unable to adjust for this bias because none of the articles reported submitting a random sample of patients with normal noninvasive tests to angiography. In general, correction for verification bias increases the sensitivity and decreases the specificity of the diagnostic test. Third, we did not account for the ability of the noninvasive test to detect other morphologic features of the atherosclerotic lesion beyond degree of stenosis. For example, carotid duplex ultrasonography, Doppler ultrasonography, and magnetic resonance angiography can, but oculoplethysmography cannot, detect ulcerative lesions within the atherosclerotic lesion.

Although we were unable to correct for verification bias, we did assess the effect of several key qualitative factors in the articles. The 70 studies included in this analysis are those of the 568 articles screened for this study that used the most rigorous methods. To be included, studies had to subject all patients to the criterion standard study of angiography, and we had to be able to recreate contingency tables with specified degrees of stenosis for both the noninvasive test results and the angiography results. We determined that in 84% (n = 59) of the remaining studies, interpretation of the reference standard was blind to the reference standard results. The results of the meta-analysis did not differ substantially from reported results when our analysis was limited to these studies. For example, the effectiveness scores for Doppler ultrasonography calculated from studies in which the noninvasive tests were interpreted blind to angiography results were 3.4 for 100% stenosis, 2.7 for greater than 70% stenosis, and 2.7 for greater than 50% stenosis. These values are all slightly higher than the effectiveness scores for the entire sample of studies (Figure 5). The CIs for both groups overlap. We also divided the studies into those in which the patients were assembled from convenience samples and those in which they were assembled from consecutive samples. Again, the effectiveness scores for the two groups of studies did not differ systematically or significantly. We believe that the lack of difference in the outcomes of the meta-analysis is attributable to the higher quality of the studies included in our study.

The information reviewed here suggests that four noninvasive tests (carotid duplex ultrasonography, carotid Doppler ultrasonography, magnetic resonance angiography, and supraorbital Doppler ultrasonography) provide useful screening information about potential candidates for endarterectomy. The finding of occlusion on noninvasive tests should not preclude further evaluation with angiography. At this point, noninvasive testing does not appear to be a sufficient substitute for angiography for patients about to have carotid endarterectomy. To supplant angiography, new noninvasive strategies, including combinations of tests, must be able to distinguish occlusion from tight stenosis and to identify the presence of intracranial disease.