A more appropriate indicator of a trainee's technical competence would be that person's ability to successfully complete a particular procedure, regardless of the number of procedures that he or she has done. Cass and colleagues [4] used this approach for upper endoscopy and colonoscopy. With a reported morbidity rate of 7% to 10% [5-8] and a mortality rate as high as 1.2% [5, 6, 8], ERCP has one of the highest complication rates among procedures routinely done by gastrointestinal endoscopists. The procedure is also technically demanding. No scientific data exist to guide the development of training guidelines for ERCP. The resources needed to train physicians in ERCP are unknown. We therefore studied the learning curve for ERCP by prospectively evaluating gastroenterology fellows at a tertiary medical center.

Methods

Top Methods Results Discussion Author & Article Info References

All ERCPs done by gastroenterology fellows at Duke University Medical Center between July 1993 and June 1995 were included in the study sample. Seventeen gastroenterology fellows were evaluated at various stages of training. If a fellow had done ERCP before the study began, this previous experience was considered in the calculation of the number of procedures done. Ten fellows had not previously done ERCP, and 7 had done a median of 73 ERCPs (range, 50 to 100 ERCPs) before the study began.

Fellows were encouraged to read basic textbooks on endoscopic technique and to attend regular clinical conferences on endoscopy and clinical issues pertinent to hepatobiliary and pancreatic disorders. Seven faculty members (attending physicians) supervised the fellows during the study period.

The fellow initiated each procedure under the supervision of an attending physician. The fellow was permitted to continue the procedure for as long as he or she was safely making progress. The attending physician intervened if the patient showed substantial or sustained discomfort, if the fellow failed to make progress, or if the procedure was unduly prolonged. The decision on when to intervene because of lack of progress was based on the attending physician's assessment of whether or not the maneuvers being done by the fellow were appropriate and likely to be successful. In general, the attending physician would allow a fellow who was executing the procedure correctly about 10 minutes to achieve a particular technical end point before intervening. If the attending physician did intervene, he or she decided whether or not to allow the fellow to attempt a subsequent component of the examination, such as stent insertion.

Immediately after the procedure, the attending physician scored the fellow on various technical components of the examination and on overall skill. The physician also determined the percentage of the total time of the ERCP that was done by the fellow. All scores were grouped in blocks of 20 ERCPs. This grouping was done because not every technical component of the examination was attempted during every procedure. Selecting blocks of 20 ERCPs provided an acceptable number of data points for most technical components.

Each fellow was graded on the following specific technical components: cholangiography, pancreatography, deep cannulation of the common bile duct, deep cannulation of the pancreatic duct, stent insertion, sphincterotomy, and stone extraction. Technical components were graded on a scale of 1 to 5 (1, excellent; 2, adequate; 3, partially successful; 4, failed; 5, no attempt). The fellow was given a rating of "acceptable" if he or she scored 1 or 2 and was given a rating of "failed" if the score was 3 or 4. In the latter situation, the attending physician graded his own performance as successful, failed, or no attempt. Adequate skill in a particular technical component of the examination was arbitrarily defined as reflecting competency if the probability of an acceptable score was 0.8 or greater for each group of 20 ERCPs and remained at this level for all subsequent ERCPs. For the final two groups of 20, the probability had to be 0.8 or greater for both. The 0.8 probability level was chosen arbitrarily because of the lack of any data indicating a suitable level. The experienced attending physicians considered this probability to be an appropriate success rate for fellows completing training in ERCP. After our study began, the American Society for Gastrointestinal Endoscopy independently decided to use a similar definition [3].

The scoring for overall skill (Table 1) was adapted from the grading system used previously in a study of competency in flexible sigmoidoscopy [9]. A score of 1, 2, or 3 indicated competence for that particular ERCP only. We decided a priori that overall competence for ERCP would be reflected by a score indicating competence (1, 2, or 3) in all ERCPs in a particular block of 20. Attending physicians were given clear instructions stating that inability to achieve a specific technical component of the examination did not in its own right constitute lack of competence. A fellow who failed one or more components of the examination could still receive an overall score of 1, 2, or 3 if the attending physician judged the procedure to have been technically complex and difficult. All data on the scoring sheet were subsequently entered into a computerized database for analysis.

Before the study, all attending physicians were instructed about the aim of the study, the guidelines to use for the amount of hands-on time the trainee would be allowed, and the manner in which the trainee would be scored. The attending physicians were told that to achieve a score of 1 (excellent) or 2 (adequate) for a technical component of the ERCP procedure, the fellow had to complete that component without assistance. Each attending physician was given the definitions for the grading of overall competence.

The fellows entered the results of all ERCPs into the standard endoscopy database used in the endoscopy unit. These data were used to provide information about the types of ERCP done during the study.

All data were analyzed in groups of 20 ERCPs. The technical components of the examination were analyzed as the probability of achieving an acceptable score (1 or 2) for all fellows within the block of 20. The 95% CIs were calculated for all data points. The scores for overall skill were analyzed as the mean ± SE of the mean for each group of 20. The probability and 95% CI of achieving an acceptable score for overall skill (grade of 1, 2, or 3) were also analyzed. A generalized linear mixed-models procedure [10], which allowed a varying number of repeated binary measures for several persons, was used for predicting the number of ERCPs that would be required to reach specified levels of competence in overall skill.

Results

Top Methods Results Discussion Author & Article Info References

Between 1 July 1993 and 30 June 1995, 1796 ERCPs were done. The attending physicians completed evaluation forms for 1450 ERCPs (81%). All data were collected prospectively. Forms were not completed when the attending physician forgot or was too busy. The median number of ERCPs done by the fellows was 132 (range, 57 to 186 ERCPs). The numbers of fellows evaluated within each block of 20 ERCPs were 10 for fellows who had done 0 to 80 and 101 to 120 procedures, 9 for those who had done 81 to 100 and 121 to 140 procedures, 7 for those who had done 141 to 160 procedures, 5 for those who had done 161 to 180 procedures, and 3 for those who had done 181 to 200 procedures.

Analysis of demographic characteristics for patients in whom the ERCPs were done during the study period showed that 155 (8.6%) had previously had a failed ERCP, 76 (4.2%) had pancreas divisum, 78 (4.3%) had had biliary or pancreatic manometry, and 239 (13.3%) had had biliary sphincterotomy.

The results of the evaluation of fellows' technical skill for cholangiography, pancreatography, deep common bile duct cannulation, and deep pancreatic duct cannulation are shown in Figure 1. The numbers of ERCPs done before fellows achieved adequate skill (as defined in the Methods section) were 160 for cholangiography, 140 for pancreatography, and 160 for deep cannulation of the pancreatic duct. After performing these quantities of ERCPs, the probability of achieving an acceptable score was 0.73 (95% CI, 0.65 to 0.81) for cholangiography, 0.75 (CI, 0.66 to 0.83) for pancreatography, and 0.82 (CI, 0.64 to 1.00) for deep cannulation of the pancreatic duct. Fellows achieved adequate skill after 60 ERCPs for stent insertion (probability, 0.81 [CI, 0.69 to 0.93]) and after 120 ERCPs for stone extraction (probability, 0.59 [CI, 0.35 to 0.82]). The number of ERCPs required to achieve competent deep cannulation of the common bile duct and biliary sphincterotomy cannot be determined with certainty because the upper limit of the 95% CI of the probability of achieving an acceptable score is below the cut-off of the 0.8 probability for one of the last two data points. The probability of achieving an acceptable score was 0.65 (CI, 0.53 to 0.78) for deep cannulation of the common bile duct after 180 ERCPs and 0.36 (CI, 0.08 to 0.65) for sphincterotomy after 160 ERCPs.

View larger version (35K): [in this window] [in a new window]   Figure 1. Probability (95% CIs) of achieving an acceptable score for cholangiography, pancreatography, deep common bile duct cannulation, and deep pancreatic duct cannulation. The probability reflects the chance of an acceptable score (a score of 1 [excellent] or 2 [adequate]) for all fellows grouped according to the number of endoscopic retrograde cholangiopancreatographies (ERCPs), in blocks of 20, that each fellow had done. In each part of the figure, n refers to the number of ERCPs in which the respective intervention was done.

The mean overall scores are shown in Figure 2. Three fellows did 180 to 200 procedures. Within this range, all three fellows achieved an acceptable score in the 7 ERCPs they performed during the study period. Figure 2 also shows the predicted probability of a fellow achieving an acceptable score. On the basis of this curve, 137 ERCPs would be required to reach a 0.8 probability of achieving an acceptable overall score and 185 ERCPs would be required to reach a 0.9 probability.

View larger version (13K): [in this window] [in a new window]   Figure 2. The mean (± SE) overall score and the predicted probability of achieving an acceptable overall score. The probability reflects the chance of an acceptable score (defined as an overall score of 1, 2, or 3, signifying overall competence) for all fellows grouped according to the number of endoscopic retrograde cholangiopancreatographies (ERCPs), in blocks of 20, that each fellow had done. In the part of the Figure thatshows the mean overall score (left), n refers to the number of ERCPs for which an overall score was given.

The percentage of time that a fellow contributed to the procedure, as estimated by the attending physician, was 46% (CI, 42% to 50%) after 20 ERCPs, 69% (CI, 65% to 73%) after 100 ERCPs, and 80% (CI, 75% to 85%) after 180 ERCPs.

The attending physicians' failure rate for the technical components of the ERCP when the fellow was either partially successful or failed (score of 3 or 4) was 13.1% for cholangiography, 10.3% for pancreatography, 11.8% for deep cannulation of the common bile duct, 19.8% for deep cannulation of the pancreatic duct, 12.6% for stent insertion, 2.9% for sphincterotomy, and 15.2% for stone extraction. The mean (±SD) overall score, by attending physician for all the ERCPs that physician supervised, ranged from 2.1 ± 0.70 to 3.6 ± 0.97.

The probability of achieving an acceptable overall score was analyzed for each fellow. Overall competence was reached after 120 ERCPs for two fellows (probability, 0.95 [CI, 0.85 to 1.00]) and after 140 ERCPs for two additional fellows (probabilities, 0.84 [CI, 0.68 to 1.0] and 1.0). After 140 ERCPs, five fellows had not achieved overall competence.

Discussion

Top Methods Results Discussion Author & Article Info References

In this prospective evaluation of ERCP training, the competence of fellows at a tertiary referral medical center was studied during a 2-year period. Each fellow was graded on the ability to successfully complete various technical components of ERCP and on overall competence in ERCP.

Our results indicate that at our medical center, fellows must have experience with at least 180 ERCPs to obtain an overall score that is consistent with competency. Only three fellows reached this level of experience. We therefore used a statistical procedure [10] that considered the data from the scores of all fellows throughout their training experience to predict the number of ERCPs that would be required for overall competence (Figure 2). The predicted probability of obtaining an overall score indicating competency was 0.9 after 185 ERCPs.

We also evaluated success in completing specific technical components of ERCP. Proficiency in deep cannulation of the common bile duct and sphincterotomy were not achieved in the study group as a whole because the upper limit of the 95% CI did not consistently include the 0.8 probability level. Deep cannulation of the common bile duct and sphincterotomy are important technical components of ERCP, and the former must be completed successfully in order to do most therapeutic interventions. This concept is supported by the finding that 60 ERCPs were required before a fellow could successfully do stent insertion. Stent insertion, however, cannot be done without deep cannulation of the common bile duct. If the attending physician in our study was required to do deep cannulation of the common bile duct when a fellow failed, the attending physicians could still let the fellow attempt the subsequent stent insertion, for which he or she would receive a score. This technique is relatively easy, as confirmed by the observation that after doing 60 ERCPs, the fellows could successfully insert the stent.

Data for individual fellows show that two fellows had reached overall competence after 120 ERCPs, whereas five had not yet done so after 140 ERCPs. The two fellows who learned quickly were, from the outset, planning careers in therapeutic endoscopy. One of the five fellows who did not reach competence after 140 ERCPs had planned a career in therapeutic endoscopy.

Competence in ERCP has received little attention in the literature, but several reviews and editorials have discussed the pertinent issues [1, 11-14]. The definitions we used for competence are our own. Are they reasonable? The definition of competence in gastrointestinal procedures has traditionally been based, at least in part, on the number of procedures done [2]. This definition, however, seems to be an inadequate surrogate marker for competence. Until recently, the American Society for Gastrointestinal Endoscopy recommended that trainees do at least 100 colonoscopic procedures before their competency is assessed [2]. Data from Cass and colleagues [4] and from Marshall [15] suggest that these numbers may be inadequate. In Cass and colleagues' study [4], an evaluation process similar to ours was used to prospectively evaluate trainees doing colonoscopy and upper endoscopy. The investigators found that trainees needed to do more than the threshold numbers of procedures recommended by the American Society for Gastrointestinal Endoscopy. These data support our finding that more extensive training is required to achieve competence in endoscopic procedures. Although the American Society for Gastrointestinal Endoscopy stated from the outset that the numbers they recommended were the minimum needed before competency could be assessed, use of this kind of threshold generated more controversy than benefit [11]. The Society for American Gastrointestinal Endoscopic Surgeons, for example, chose not to observe these thresholds because it considered them excessive [16]. The current American Society for Gastrointestinal Endoscopy recommendations for advanced endoscopic training (including ERCP) [3], published after our study began, no longer suggest minimum numbers of procedures for competence. Instead, technical competence in ERCP has been redefined along the lines suggested in our study. The guidelines state that "it is reasonable to expect graduates of advanced training programs to reliably (at least an 80% success rate) obtain access to (selectively and freely cannulate) the desired duct, without assistance" [3]. According to this definition, the numbers required to achieve competence in our study are considerably greater than the previous recommendation of a minimum of 100 ERCPs for assessing competency [2]. They are also significantly greater than the median number of 50 that practicing gastroenterologists believe is required to attain competence in ERCP [17].

Although no gold standard is available for defining competence, we believe that success rates are a move in the right direction. The success rate selected can be individualized for each procedure. For example, pyloric intubation during upper endoscopy should be done successfully in almost 100% of procedures before a trainee is considered competent. In ERCP, a procedure that is more technically demanding than pyloric intubation, the expectation for competence must be adjusted downward. The Figure of 80%, selected for our study, is in agreement with that selected by the American Society for Gastrointestinal Endoscopy [3].

Our data should be interpreted with caution. For example, we examined technical competence in ERCP, a measure that should not be extrapolated to overall physician competence [1, 2]. Because only three fellows had done more than 180 ERCPs, it is difficult to firmly recommend an absolute number of procedures that should be required for competency. Grading of the overall score is susceptible to bias. The mean (±SD) score for each attending physician across all ERCPs varied from 2.1 ± 0.70 to 3.6 ± 0.97. We did not specifically control for interobserver bias, but each attending physician supervised all ERCPs for 1 week at a time. Therefore, the effect of an attending physician whose scores were higher or lower than the scores of the other attending physicians should be spread over all fellows. Similarly, the potential bias associated with the inherently subjective nature of the criteria in Table 1 should be reduced if the attending physician was consistent in his scoring criteria. This scoring system has not been formally validated but is a modification of a scoring system used for training in flexible sigmoidoscopy [9]. Another potential source of bias is that the attending physician could not be blinded to the fellow's previous experience; when choosing the score, however, the attending physician was unaware of the actual number of procedures each fellow had done. The probability of success for the technical components of ERCP is a particularly useful grading system: It is less susceptible to bias because it is scored as either a success or a failure. The acquisition of skill in these technical components was similar to the attainment of competence for overall skill. This finding seems to support the validity of the data for overall skill.

Our study was not designed to address such controversial issues as who should receive training and whether self-instruction has any role in a complex procedure such as ERCP. The study does, however, highlight the fact that substantial numbers of procedures are required to train fellows adequately. Individual fellows may learn more quickly than a group as a whole, a hypothesis that is confirmed by the fact that two fellows achieved overall competence after 120 ERCPs. Because no method is yet available to predict which fellows will learn quickly, ERCP should only be taught in centers that have sufficient resources (including preceptors and patient volume) to ensure that a comprehensive training experience can be provided [11]. Conversely, training programs that lack these resources may be unable to support a training program in ERCP. Our own training program is being restructured to give fewer fellows more intense training.

The implications for training guidelines and for the allocation of resources are substantial. Procedural competence is an important issue for medicolegal reasons [12] and in maintaining high standards of patient care. Of all procedures routinely done by gastroenterologists, ERCP has the highest complication rate. In a recent large study [18], complications of endoscopic sphincterotomy were shown to be related to endoscopic technique; the latter, in turn, was shown to be associated with the number of procedures done. Although not evaluated in the study, technique may also be a function of the skill and training of the endoscopist. The medical community has an obligation to ensure that ERCP is done by competent physicians. In addition, reducing the number of ERCPs that are "failed" or "inadequate" because they are done by endoscopists who have an unacceptably low success rate will help contain medical expenditure [11].

The methods we used to assess competence were based on whether or not intervention was successful. This concept can be used to assess educational outcomes beyond gastrointestinal endoscopy. A similar evaluation process could be used for procedures ranging from lumbar puncture by the internist to cardiac catheterization by the cardiologist. Evidence-based assessment is an effective way to combat assumed competence in medical procedures. The assumption that a physician trained in a diagnostic procedure can safely perform a more complex therapeutic procedure is dangerous and has no place in modern medical practice. Unfortunately, assumed competence is not infrequent in the procedure-oriented medical specialties.

In summary, our study provides insight into the learning curves of gastroenterology fellows training in ERCP and indicates the number of ERCPs needed for competence. Guidelines should be developed to indicate what constitutes an acceptable success rate; this rate should guide the certification process for technical competence. Training programs that lack the necessary human and technical resources and number of procedures cannot and should not offer fellows certifiable training in diagnostic and therapeutic ERCP. Our study also shows the difficulty of correlating numbers of completed procedures and competence. Methods of assessing competence in medical procedures should be carefully evaluated on an ongoing basis. Use of evidence-based standards for competence, such as success rates, is ideal because it considers the various learning curves of individual trainees.

Dr. Baillie: Box 3189, Duke University Medical Center, Durham, NC 27710.

Dr. Branch: Box 3662, Duke University Medical Center, Durham, NC 27710.

Dr. Affronti: Gastroenterology Division, University of Florida Health Science Center, 653 West 8th Street, Jacksonville, FL 32209.

Ms. Browning: Box 3423, Duke University Medical Center, Durham, NC 27710.

Dr. Bute: Box 3830, Duke University Medical Center, Durham, NC 27710.