Reliability of Laparoscopic Skills Assessment on Video: 8-Year Results of the Endoscopic Surgical Skill Qualification System in Japan

Matsuda, Tadashi; Kanayama, Hiro‐omi; Ono, Yoshinari; Kawauchi, Akihiro; Mizoguchi, Hiroaki; Nakagawa, Ken; Shigeta, Masanobu; Habuchi, Tomonori; Terachi, Toshiro

doi:10.1089/end.2014.0092

Cited by 22 publications

(12 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They stated that long-term experience with their assessment method may be necessary to be able to perform reliable skill assessments. 18 The results of this study might likewise indicate that the interrater reliability is jeopardized when GOALS is used without proper instructions and training of the raters.…”

Section: Discussionmentioning

confidence: 82%

Validity and Reliability of Global Operative Assessment of Laparoscopic Skills (GOALS) in Novice Trainees Performing a Laparoscopic Cholecystectomy

Kramp

Det

Hoff

et al. 2015

Journal of Surgical Education

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 82%

Validity and Reliability of Global Operative Assessment of Laparoscopic Skills (GOALS) in Novice Trainees Performing a Laparoscopic Cholecystectomy

Kramp

Det

Hoff

et al. 2015

Journal of Surgical Education

View full text Add to dashboard Cite

“…After the training session, completed questionnaires were collected including demographic data, experience of laparoscopic surgeries and simulation training. In Japan, in 2004, the ESSQ system was initiated, in which two double‐blinded referees evaluate an unedited surgical movie 12,13 . Information on this ESSQ qualification status was also collected.…”

Section: Methodsmentioning

confidence: 99%

Development and validation of a porcine organ model for training in essential laparoscopic surgical skills

et al. 2020

View full text Add to dashboard Cite

Objectives: To develop a wet laboratory training model for learning core laparoscopic surgical skills and evaluating learners' competency level outside the operating room. Methods: Participants completed three tasks (task 1: tissue dissection around the aorta; task 2: tissue dissection and division of the renal artery; task 3: renal parenchymal closure). Each performance was video recorded and subsequently evaluated by two experts, according to the Global Operative Assessment of Laparoscopic Skills and taskspecific metrics that we developed (Assessment Sheet of Laparoscopic Skills in Wet Lab score). Mean scores were used for analyses. The subjective mental workload was also assessed (NASA Task Load Index). Results: The 54 participants included 32 urologists, eight young trainees and 14 medical students. A total of 13 participants were categorized as experts (≥50 laparoscopic surgeries), eight as intermediates (10-49) and 33 as novices (0-9). There were significant differences in the Global Operative Assessment of Laparoscopic Skills and Assessment Sheet of Laparoscopic Skills in Wet Lab scores among the three groups in all three tasks. Higher NASA Task Load Index scores were observed in novices, and there were significant differences in tasks 1 (Kruskal-Wallis test, P = 0.0004) and 2 (P = 0.0002), and marginal differences in task 3 (P = 0.0745) among the three groups. Conclusions: Our training model has good construct validity, and differences in the NASA Task Load Index score reflect previous laparoscopic surgical experiences. Our findings show the ability to assess both laparoscopic surgical skills and mental workloads, which could help educators comprehend trainees' level outside the operating room. Given the decreasing opportunity to carry out pure laparoscopic surgeries because of the dissemination of robotic surgery, especially in urology, our model can offer practical training opportunities.

show abstract

“…In our earlier research with GOALS assessment, we found a relatively low reliability compared to other studies [ 11 ]. We hypothesized that the lack of exposure and/or training to the assessment method might be one of the contributing factors, as was seen in a series of other studies [ 6 , 12 , 13 ]. In this study, the video assessments were therefore preceded by an introduction in order to calibrate the raters in the following way: (1) The items on the assessment forms were explained, (2) raters were encouraged to use the full scales as much as possible, (3) raters were instructed to use their own opinion when rating with the independence-scaled procedural assessment, and (4) we attempted to calibrate the raters by giving a clear definition of the low and high end of the scale of the GRSs items with a 2-min operative videos of a novice ( N = 1) and of a consultant surgeon ( N > 100).…”

Section: Methodsmentioning

confidence: 99%

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

et al. 2015

View full text Add to dashboard Cite

BackgroundThere is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs).MethodsAn independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion.ResultsIndependence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692).Discussion/conclusionA procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.

show abstract

Reliability of Laparoscopic Skills Assessment on Video: 8-Year Results of the Endoscopic Surgical Skill Qualification System in Japan

Abstract: The ESSQ system showed moderate reliability for the video assessments by the referees. It was concluded that the video assessments by the referees were fair for all applicants, because the final qualification rates showed no significant differences among the referees.

Cited by 22 publications

References 12 publications

Validity and Reliability of Global Operative Assessment of Laparoscopic Skills (GOALS) in Novice Trainees Performing a Laparoscopic Cholecystectomy

Validity and Reliability of Global Operative Assessment of Laparoscopic Skills (GOALS) in Novice Trainees Performing a Laparoscopic Cholecystectomy

Development and validation of a porcine organ model for training in essential laparoscopic surgical skills

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Contact Info

Product

Resources

About