Methods

Top Methods Results Discussion Author & Article Info References

Patients and Procedures

Patients and health care personnel who were in an internal medicine department and needed a tuberculin skin test were invited to participate. Persons who had received bacille Calmette–Guérin vaccine were enrolled preferentially. Ninety-six persons who provided informed consent ultimately participated in the study.

Tuberculin Skin Tests and Measurement Methods

Ten units of tuberculin from Pasteur Merieux, Lyon, France (corresponding to the recommended 5 IU of purified protein derivative tuberculin), were injected intradermally on the volar surface of the forearm (Mantoux technique) [11]. Readings were done on the third day after the test was administered, and the diameter of induration was measured along the long axis of the forearm. Two experienced investigators each independently did three measurements. The first two measurements were taken with a blinded caliper using the ballpoint-pen technique [3]. With this technique, a medium-point ballpoint pen is used to draw a line starting 1 to 2 cm away from the skin reaction and moving toward its center. When the pen reaches the margin of the induration, an increased resistance to further movement is felt and the pen is lifted. The procedure is repeated on the opposite side of the skin reaction. The distance between the ends of the opposing lines at the margins of the induration is measured. In our study, the lines were erased and the measurement process was repeated. The lines were then erased again, and the third measurement was done by palpation [2]. To reproduce the usual conditions of testing, we used a flexible ruler.

The data were collected during eight sessions; 11 to 14 participants were tested per session. To reduce the chance that an observer would remember previous readings, three things were done. First, the results of measures that were obtained with the blinded caliper were recorded by a third investigator. Second, the first ballpoint-pen measure was done for all participants at each session, then the second ballpoint-pen measure, and then the palpation measure. Third, before the second and third readings, the third investigator verified that no minor landmarks persisted.

Statistical Analysis

To analyze the reliability of quantitative data, we used statistical methods that have been described elsewhere [12]. Intraclass correlation coefficients and their 95% CIs were computed using SAS soft-ware (SAS Institute, Cary, North Carolina) [13]. Induration diameters were used to classify skin reactions as positive or negative according to the 5-, 10-, and 15-mm cutoff points that have been recommended as indicating positivity in various situations [2]. Reliability was then assessed with coefficients [14]. We also used a graphical analysis that focuses on the mean and the variation in the differences between repeated measurements [15]. Mean differences and the SD of the differences were calculated. An area of imprecision that was determined on the basis of the SD of the differences was placed around the arbitrarily chosen 10-mm cutoff value (10 mm ± 1.96 SD). If a first measurement fell within this area, particularly at or about the cutoff value, the likelihood that the second measurement would be sufficiently different to change the result of the tuberculin skin test from negative to positive (or vice versa) was high. Conversely, such reclassification would occur in only 5% of the cases that had values outside this area.

Results

Top Methods Results Discussion Author & Article Info References

Because of the study design, only 27 participants (28%) did not react to the tuberculin skin test.

Reliability of the Ballpoint-Pen Technique

Intraobserver Reliability

In persons who had no response to the tuberculin skin test, the intraobserver reliability was perfect (intraclass correlation coefficient = 1.0). Intraclass correlation coefficients were high for both observers and decreased only slightly after the nonresponders were excluded. The coefficients also suggested good intraobserver reliability but were lower with the 10- and 15-mm cutoff values than with the 5-mm cutoff value (Table 1).

The top panel of Figure 1 shows the difference between the two readings for each participant that were done by the first observer (range, –6.8 to +3.5 mm) plotted against the corresponding mean for each participant. The level of intraobserver reliability was evaluated by determining the 95% CI ( –2.68 to +2.96 mm) within which most of the differences were seen. This means that 5% of the time, the second measure of the test results done by using the ballpoint-pen method would be at least 2.7 mm less than or 3.0 mm more than the first one. This lack of reliability could lead to the reclassification of a negative tuberculin skin test result as positive or vice versa.

View larger version (18K): [in this window] [in a new window]   Figure 1. Top. Intraobserver reliability for the ballpoint-pen technique. Differences between the repeated measurements taken by observer 1 (M1Obs1 and M2Obs1) that were obtained using the ballpoint-pen technique are plotted against the mean of repeated measures. Middle. Interobserver reliability for the ballpoint-pen technique. Differences between the first measurement taken by each observer (M1Obs1 and M1Obs2) that were obtained using the ballpoint-pen technique are plotted against the mean of these first measures. Bottom. Interobserver reliability for the palpation technique. Differences between the measurement taken by each observer (Obs1 and Obs2) that were obtained using the palpation technique are plotted against the mean of these measurements. The horizontal lines in all panels indicate the mean difference (thick line) ± 2 SDs (thin lines). The closed circle with an asterisk in each panel corresponds to the superposition of the measures of 27 persons. Areas of imprecision (bars) that were derived from the SDs of the differences were defined for the 10-mm cutoff point for all comparisons. If a first measurement falls within this area, particularly at or about the cutoff value, a second measurement would probably be sufficiently different to lead to reclassification of a negative result of the tuberculin skin test as positive or vice versa. Conversely, such reclassification would occur in only 5% of the cases whose values lie outside this area. Open circles represent patients for whom the tuberculin skin test result was reclassified after the second measurement (all had a first measure that fell in the area of imprecision). Open circles with a number correspond to the superposition of two or four patients. The s in the top and middle panels represent patients whose test results were reclassified after the second measurement even though their first measurement was outside the area of imprecision.

As shown in the top panel of Figure 1, an area of imprecision that straddles the cutoff value (7.2 to 12.8 mm for a 10-mm cutoff value) was generated using the SD of the differences. Test results for 8 of the 69 patients (12%) were reclassified. The first measurement for 30 of the 69 patients (43.5%) fell within this area of imprecision; 7 of those 30 patients (23.3%) were among the 8 patients whose test results were reclassified.

Interobserver Reliability

Agreement between observers, estimated by using the intraclass correlation and coefficients, was high (Table 1). The first ballpoint-pen measures made by the two observers were used for these analyses.

Differences between first measures done by the two observers were between –5.1 and +7.3 mm (Figure 1, middle). The 95% CI of the differences was –3.39 to +3.69 mm; this means that 5% of the time, the result of a second tuberculin skin test measurement by another investigator would be at least 3.4 mm more than or 3.7 mm less than that of a first investigator. As in the top panel of Figure 1, an area of imprecision (6.5 to 13.5 mm) is shown in the middle panel of Figure 1; this area is slightly broader than that calculated for intraobserver reliability. Test results for 8 of the 69 patients (12%) were reclassified. The first measurement for 40 of the 69 patients (58%) fell within this area of imprecision; 7 of those 40 patients (17.5%) were among the 8 patients whose results were reclassified.

Reliability of the Palpation Technique

Except for the coefficients at the 15-mm cutoff, assessment of agreement between observers showed that all reliability coefficients obtained with the palpation technique were slightly lower than those obtained with the ballpoint-pen method (Table 1).

The 95% CI of the differences between the measures of the two observers was –4.6 to +5.2 mm (Figure 1, bottom). This resulted in a much broader area of imprecision for the readings (5.1 to 14.9 mm). Test results were reclassified for 12 of the 69 patients (17.4%). The first measure of 43 of the 69 patients (62.3%) fell within this area of imprecision, and the 12 patients whose test results were reclassified were among those 43 (27.9%).

Agreement between Ballpoint-Pen and Palpation Methods

Although all the intraclass correlation coefficients were high, the coefficients that were produced after persons with no response to the test were excluded suggested only moderate to good reliability (Table 1). The 95% CIs of the differences between the first ballpoint-pen and the palpation measures were –3.0 to +4.1 mm for readings taken by the first observer and –2.5 to +3.9 mm for readings taken by the second observer. The areas of imprecision for the measurements were from 6.4 to 13.6 mm for readings taken by the first observer and 6.8 to 13.2 mm for readings taken by the second observer. Reclassification occurred in 8 of 69 patients (12%) for both observers.

Discussion

Top Methods Results Discussion Author & Article Info References

In our study, the ballpoint-pen technique was reliable, as evaluated by global reliability coefficients. However, the graphical analysis provided a more meaningful representation of the level of variation. Intraobserver reliability may be the most important factor for such diagnostic tests as the tuberculin skin test, which are usually done by only one examiner for any given patient. Lack of reliability may lead to the frequent reclassification of results, particularly if readings are at or about the cutoff values. Reliability coefficients were slightly higher for the ballpoint-pen technique than for the palpation method. In addition, the 95% CI of the differences of the measures taken by the two observers was 38% broader for the palpation method than for the ballpoint-pen technique; this could result in more frequent misclassification.

Only one study [10] has addressed the interobserver reliability of the ballpoint-pen technique. That study relied on simple correlation coefficients to determine reliability. Reanalysis of the data from that study provided a coefficient of 0.74 (using a cutoff point of 10 mm). Previous studies of the reliability of the palpation method [5, 6, 10] have also been restricted primarily to the assessment of interobserver agreement and have provided conflicting results. Recalculation from the data of one large survey of six studies on tuberculin skin testing [4] gave slightly wider CIs for the differences between readings by two observers (range, –3.1 to +3.3 mm to –4.2 to +5.0 mm) than between two readings from the same observer for the difference (range, –2.7 to +3.3 mm to –3.5 to +4.5 mm). Four studies [7-10] compared the ballpoint-pen and palpation methods and had discordant results. None of these studies used reliability coefficients, which are the generally recommended measures of agreement.

Recall bias would make our findings more important if it were present, because it would tend to improve repeatability. Because the tuberculin reaction could not be measured by a blinded technique, we cannot exclude an unconscious bias that would have favored the ballpoint-pen method. In addition, knowledge of the standard cutoff values may have influenced the observers when they used the ruler. Finally, the generalizability of our study results is limited because only two observers did the measurements, both of whom have considerable experience reading tuberculin skin test indurations. Persons who occasionally read induration from tuberculin skin tests may not have as much experience as these two observers have.

Results of tuberculin skin tests influence the decision to initiate antituberculous therapy [16-18]. Although errors in classification cannot be completely avoided, they should be minimized by the implementation of appropriate rules for measurement. Our study suggests that current recommendations for tuberculin skin testing should be reconsidered. We propose that induration be measured with the ballpoint-pen technique. Clinicians should be more cautious when interpreting a tuberculin skin test result that is close to the cutoff value than they would be when interpreting one that is far removed. To improve intraobserver reliability, the final result could be the average of two consecutive measures, preferably taken with a blinded gauge. Even when done in the most reliable way possible, however, the tuberculin skin test remains an imperfect diagnostic tool and should not replace clinical judgment.

Dr. Coste: Departement de Biostatistiques, Hopital Cochin, 75674 Paris Cedex 14, France.

Dr. Esdaile: Mary Pack Arthritis Centre and Vancouver Hospital, University of British Columbia, 895 West 10th Avenue, Vancouver, British Columbia V5Z 1L7, Canada.