The goal of this study was to relate the acquisition of a few basic skills to the number of procedures using a simple, quick, and objective grading system for each procedure.

Methods

Top Methods Results Discussion Author & Article Info References

Trainees were proctored at the gastrointestinal laboratory at Hennepin County Medical Center, a 520-bed county hospital. The three-bed gastrointestinal laboratory does approximately 1000 esophagogastroduodenoscopies and 350 colonoscopies per year. All endoscopic procedures are approved by the staff, using the indications in the consensus statement of the American Society for Gastrointestinal Endoscopy [7]. The case mix of these procedures is typical of most endoscopic practices except that 20% of all procedures are done in an intensive care unit on seriously ill patients and about 10% involve gastrointestinal bleeding. Over 90% of procedures are done by a trainee, and all procedures were under the guidance of one of four staff gastroenterologists or one staff surgeon. Seven gastroenterology fellows in their first through third years of endoscopic training and five fourth-year surgery residents in their first year of endoscopic training did consecutive elective and emergency procedures without any conscious attempt to limit their access to any type of procedure. Difficult therapeutic measures (sclerotherapy, dilatation of achalasia, therapy of bleeding ulcers) after the diagnostic endoscopy were done by the proctor. Trainees did biopsies and cauterized and snared polyps. Trainees were graded only on procedures done during their time at Hennepin County Medical Center. The number of proctored procedures done elsewhere as part of training was taken from each trainee's logbook.

Trainees were encouraged to review videotapes on esophagogastroduodenoscopy and colonoscopy and were provided access to textbooks and atlases on these subjects. They attended a once-monthly endoscopy conference at which they reviewed endoscopic slides with staff and gave reviews of a topic in gastrointestinal endoscopy.

Training was primarily done using videoendoscopes. Endoscopic light sources were plugged into an outlet strip to which a timer was attached. This system allowed automatic assessment of procedure duration (measured from the time the light source was switched on until it was switched off). Procedure duration was recorded as part of the endoscopic report. During the procedure, the proctors were instructed to teach as they normally would but to not tell the trainee whether an abnormality was present until the trainee had stated his or her opinion.

After each procedure, proctors used a menu-driven PASCAL computer program to record the trainees' performance. After esophagogastroduodenoscopy, the proctors noted whether the trainee could intubate the esophagus, traverse the pylorus, and correctly identify any major abnormalities. After colonoscopy, they noted whether the trainee could intubate the splenic flexure, intubate the cecum (as recognized by either the appendix or ileocecal valve) without assistance, and recognize any major abnormalities. If anatomic changes (such as a right hemicolectomy) made a task impossible, if the trainee was not permitted to do a task (usually a known difficult esophageal intubation), or if the staff could also not do a task (usually intubating the cecum), a grade of "not applicable" was recorded. Data were stored on a microcomputer using dBase III+ (Ashton-Tate, Torrance, California). The time required to enter a grade was automatically recorded by the computer program. For the first 6 months of this 27-month study, proctors were encouraged to grade trainees. Compliance with grading, however, was poor; proctors would leave in the middle of a grading session and not complete the report until hours later. The median time required to grade endoscopies was 22 seconds (range, 2 to 16 161 seconds), and the median for colonoscopies was 9.5 seconds (range, 3.1 to 6803 seconds). The longer times were recorded when the proctor left in the middle of grading, often not returning until hours later. For the first 6 months, when grading was optional, a mean of only 70% of procedures were graded. After grading was linked to the production of an endoscopic report, compliance was 100%. The computer program used to produce endoscopic reports was subsequently modified so that an endoscopic report was not produced for any procedure done by a trainee unless the trainee was graded. Immediate complications were prospectively recorded as part of the computer-generated endoscopic report and stored using dBase III+. Remote complications were entered in a logbook.

The "gold standard" in each case was that which the attending physician could do and what he or she saw. Proctors had a median of 8 years (range, 3 to 26 years) of experience doing endoscopies. A retrospective review of the endoscopic procedures done by the proctors alone over a 2-year period was conducted to determine how often the proctors could intubate the esophagus and reach the cecum.

As the trainees became more accomplished (at a mean of 6 months of training), they were permitted to record their own grades in the computer after receiving them from the proctor. They were told that the data they entered would not be used to judge their competence but would be published as part of an effort to determine how many procedures are necessary to gain competency. The data from 15% of the procedures in which trainees did poorly were checked, and no mistakes were found. Confidence intervals (CIs) of 95% are given where appropriate.

Results

Top Methods Results Discussion Author & Article Info References

Each trainee was graded for a median of 113 esophagogastroduodenoscopies (range, 54 to 162 procedures) and 49 colonoscopies (range, 39 to 127 procedures). Procedure times did not change significantly with training. Trainees took an average of 21 minutes (95% CI, 16 to 26 minutes) to do endoscopy after the first 10 procedures and a mean of 22 minutes (CI, 13 to 31 minutes) to do the procedure after 100 procedures. They took 48 minutes (CI, 37 to 59 minutes) to do colonoscopy after 10 procedures and 55 minutes (CI, 11 to 99 minutes) after 100 procedures. Abnormality recognition was initially greater than 85% and did not change significantly over the course of the grading periods.

Esophageal, pyloric, splenic flexure, and cecal intubation rates (means and CIs) are related in Figure 1 to the trainees' previous experience. At 50 esophagogastroduodenoscopies, the trainees could intubate the esophagus a mean of 95% of the time. At 100 procedures, however, this rate decreased to 75%. This decline coincided with the introduction of more difficult patients into the trainees' schedule. These patients included those with complicated postsurgical proximal esophageal strictures.

View larger version (20K): [in this window] [in a new window]   Figure 1. Percent success at intubation of each area during the previous 10 procedures related to the number of procedures done. Data are means with 95% confidence intervals for all trainees evaluated during that training period.

To assess whether these patients were the reason for the decline in intubation, the experience of five trainees was studied in greater detail. Reasons for grades of "not applicable" recorded for esophageal intubation were noted. When trainees could not intubate the esophagus, the staff made a subjective judgment of whether intubation was easy, difficult, or impossible. Trainees who had done fewer than 40 procedures were not permitted to intubate the esophagus in 8 of 130 proctored procedures (6%). These procedures included three patients with bleeding varices, three intubated patients requiring feeding tube placement, and two patients requiring percutaneous endoscopic gastrostomy. Trainees who had done 90 to 120 procedures were not permitted to intubate the esophagus in 1 of 98 procedures (1%) (P = 0.04, one-tailed Fisher exact test). This involved an intraoperative endoscopy during which the scope was placed by the anesthesiologist. Trainees who had done fewer than 40 procedures could not intubate the esophagus in nine cases. The staff found esophageal intubation to be easy in four of these procedures (44%). Trainees who had done 90 to 120 procedures could not intubate the esophagus in nine cases. The staff found esophageal intubation to be easy in none of these procedures (0%) (P = 0.11, two-tailed Fisher exact test). In all cases, the staff were successful in intubating the esophagus. In 251 procedures done without a trainee, staff could intubate the esophagus 98% of the time (CI, 96% to 100%).

After 50 colonoscopies, trainees could pass the colonoscope to the cecum without assistance more than 80% of the time. By 100 procedures, however, they had not improved as shown in Figure 1. Single observations of fellows who had done 210 and 275 colonoscopies, respectively, showed success rates of 94% and 96%. In 65 procedures done without a trainee, staff could reach the cecum a mean of 94% of the time (CI, 90% to 98%). Data for trainees doing 70 or more colonoscopies are based on limited numbers of trainees (one or two trainees during each interval).

Performance was not clearly related to the trainee's specialty. Similarly, no individual trainee was found to be clearly more proficient than another. Procedure time, abnormality recognition, and incidence of complications did not correlate with experience. The major complication rates during the study were 0.4% for esophagogastroduodenoscopy and 0.6% for colonoscopy. Trainees and staff did not differ significantly in complication frequency, except that half of the endoscopic complications were related to continued gastrointestinal bleeding after attempts to treat actively bleeding ulcers. These procedures were done by the staff.

Discussion

Top Methods Results Discussion Author & Article Info References

Our study showed that trainees intubated the esophagus over 90% of the time after 50 procedures. When faced with a case mix representative of practice (including difficult patients), however, the intubation rate decreased and did not return to 90% until over 100 procedures had been done. At 50 colonoscopies, trainees intubated the cecum 80% of the time; however, over 100 procedures may be required to reach 90%. A case report of the acquisition of colonoscopy skills by one surgeon found that between 100 and 200 procedures were necessary before 90% cecal intubation was achieved consistently [8]. These data suggest that even the highest current estimates of procedure numbers required for competency (100 for each procedure) may be too low.

Trainees who had done fewer than 40 procedures were not permitted to attempt several esophageal intubations in difficult situations. This happened not only during the first 10 procedures but during the entire first 40 cases. When a trainee with little experience could not intubate the esophagus, it was no more likely to be difficult than easy for the proctor. In contrast, the only instance where trainees who had performed 90 to 120 procedures were not permitted to attempt intubation occurred when the scope was placed under direct vision by an anesthesiologist. When trainees were unsuccessful in intubating the esophagus, the case was always determined to be difficult. We believe that the decrease in the esophageal intubation rate seen at approximately 100 procedures occurred because proctors did not permit trainees to attempt to intubate some difficult patients during their early training. After the trainees achieved some competency, they were permitted to attempt to intubate these difficult patients. Although this practice is rational, we believe that experts' estimates of the number of procedures required to achieve competency are based on impressions of trainees' abilities to do easy procedures. The second learning curve may not have been factored into expert opinion.

We suspect that gastrointestinal endoscopy is not the only medical procedure with a decline in success rates as more difficult procedures are introduced. When should a trainee be certified as competent—when he or she has mastered the routine procedures or the difficult procedures? If the answer is the latter, are there enough such procedures to ensure competency among the number of trainees being certified? Gastroenterology societies and training directors have already suggested that only certain trainees should be taught to do difficult procedures. In this "third tier" fellowship, therapeutic endoscopy and endoscopic sphincterotomy are taught only to a few trainees who receive an extra year of training [9]. We believe that it is reasonable to require trainees to be competent in both difficult and easy esophagogastroduodenoscopy and colonoscopy procedures.

The reason that total procedure time did not change during training may have been because the proctor, early on, would finish a procedure that was difficult or was taking an inordinately long time. However, we did not record procedure difficulty, whether the proctor manipulated the scope, or the period of time that he or she took over the case. Although the frequency of recognition of specific gastrointestinal abnormalities did not change during the study, we believe that the frequency of identification of what an abnormality represented did increase. However, such data were not systematically recorded. The 85% rate of recognition that something was abnormal is similar to interobserver agreement between experienced endoscopists reported for such major lesions as duodenal ulcer (91%) [10] or deformity (78%) [10]; the presence of varices (65% to 70%) [11, 12]; or any endoscopic abnormality (68%) [13].

Our complication rate for endoscopy is higher than the 0.08% to 0.13% reported in the retrospective surveys [14, 15]. We believe that this increased complication rate was the result of the prospective data collection and the high frequency of therapeutic procedures in this series. The one series with prospective data collection reported a complication rate of 0.29% [16]. Two series of therapeutic procedures (dilatation and treatment of bleeding) reported a rate of 0.52% to 2.3% [16, 17]. Our complication rate for colonoscopy is similar to the 0.7% reported in the literature [14].

It was initially difficult to get proctors to grade trainees. Although grading took less than 30 seconds, proctors often did not do it. Reasons stated included that grading was not a part of their routine, that they were too busy that day, or that they "just forgot." Grading had to be linked to the production of an endoscopic report for compliance to approach 100%.

In its Guide for the Use of American College of Physician Statements on Clinical Competence, The Health and Public Policy Committee of the American College of Physicians states "There has been little research on the relationship between experience and competence" [18]. The American College of Physicians views competency as a combination of education, training, and experience, which lead to the acquisition of cognitive and technical skills. For gastrointestinal endoscopy, objective data relating procedure number to competency are limited and exist only for flexible sigmoidoscopy [19]. We investigated the relation between easily evaluated technical skills and procedure number. Cognitive skills of endoscopists can be tested by written tests such as the Gastroenterology Subspecialty of the American Board of Internal Medicine or the Gastrointestinal Endoscopy Self-Assessment Program of the American Society for Gastrointestinal Endoscopy. Technical skills were subjectively evaluated until the late 1980s, when graduates of gastroenterology fellowships were required to present a logbook of procedures to show that they had done a certain number of procedures [20]. However, only the most rudimentary information about the procedures was recorded.

This study has many limitations. Observations of the gastroenterology fellows' progress was not continuous because they rotated to the other teaching hospitals every 3 months. The study was also limited to participants in a single training program. Only three simple measures of technical skills were selected for each procedure. We did not measure the portion of the procedure time during which the fellow was guiding the endoscope. We did not measure whether the trainee knew what abnormalities represented. We studied only the acquisition of some technical skills and did not address the maintenance or possible diminution of competence with time.

Future studies in this area should be done at many different programs, using large numbers of trainees and proctors. Ideally, each trainee would either do all procedures at a single site or at all sites in a training program using the same grading program. Such studies would require a tremendous commitment of time, energy, and resources from all proctors and would be impossible without both funding and academic rewards. Computer simulators such as those being developed for flexible sigmoidoscopy [21], colonoscopy [22], and esophagogastroduodenoscopy and endoscopic retrograde cholangiopancreatography [23] might offer a reasonable approach to the standardization of training and the evaluation of skills.

In the end, it is important to recognize that patients (and physicians) want effective procedures to be done safely and efficiently. Effectiveness of special techniques in gastrointestinal endoscopy has been frequently studied, but the overall effectiveness of the diagnostic upper endoscopy (that is, did the examination answer the question?) has been examined only in special situations [24]. Adverse effects have usually been studied in a retrospective fashion [14, 15] until the recent interest in conscious sedation [25]; the incidence of false-positive test results has not been studied. Efficiency, and in particular, its acquisition by endoscopists, has rarely been studied. With the advent of microcomputers with the ability to record and store large quantities of objective data, it is now possible to evaluate endoscopic competence routinely in trainees and in practicing physicians.