Validity and Reliability of Global Operative Assessment of Laparoscopic Skills (GOALS) in Novice Trainees Performing a Laparoscopic Cholecystectomy

Kramp, Kelvin H.; Det, Marc J. van; Hoff, Christiaan; Lamme, Bas; Veeger, Nic J. G. M.; Pierie, Jean-Pierre E. N.

doi:10.1016/j.jsurg.2014.08.006

Cited by 56 publications

(42 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In our earlier research with GOALS assessment, we found a relatively low reliability compared to other studies [ 11 ]. We hypothesized that the lack of exposure and/or training to the assessment method might be one of the contributing factors, as was seen in a series of other studies [ 6 , 12 , 13 ].…”

Section: Methodscontrasting

confidence: 72%

“…In this study, the video assessments were therefore preceded by an introduction in order to calibrate the raters in the following way: (1) The items on the assessment forms were explained, (2) raters were encouraged to use the full scales as much as possible, (3) raters were instructed to use their own opinion when rating with the independence-scaled procedural assessment, and (4) we attempted to calibrate the raters by giving a clear definition of the low and high end of the scale of the GRSs items with a 2-min operative videos of a novice ( N = 1) and of a consultant surgeon ( N > 100). We also have hypothesized in the same study that a lack of motivation to complete a comprehensive assessment lengthy operative video material might lead to unreliable measurements [ 11 ]. Therefore, those who completed the assessments were rewarded with a box of wine of around 85$.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

et al. 2015

Self Cite

View full text Add to dashboard Cite

BackgroundThere is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs).MethodsAn independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion.ResultsIndependence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692).Discussion/conclusionA procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.

show abstract

Section: Methodscontrasting

confidence: 72%

Section: Methodsmentioning

confidence: 99%

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

et al. 2015

Self Cite

View full text Add to dashboard Cite

show abstract

“…Direct and video scores correlated significantly for OSATS, GRS, and STS, implying that video rating may be used as an alternative to direct rating. Although not as extensively used and validated, video assessment has already been done [14,15,16,17,18,19,20], while Kramp et al [21] have demonstrated that even Global Assessment of Technical Skills (GOALS) video ratings could substitute, albeit with low reliability, direct OSATS ratings. With sufficient validity, video OSATS rating could be used in place of direct OSATS rating while offering the advantages of removing rater bias and being independent of time and place restrictions.…”

Section: Discussionmentioning

confidence: 99%

Direct Observation versus Endoscopic Video Recording-Based Rating with the Objective Structured Assessment of Technical Skills for Training of Laparoscopic Cholecystectomy

Nickel

Hendrie²,

Stock³

et al. 2016

Eur Surg Res

View full text Add to dashboard Cite

Purpose: The validated Objective Structured Assessment of Technical Skills (OSATS) score is used for evaluating laparoscopic surgical performance. It consists of two subscores, a Global Rating Scale (GRS) and a Specific Technical Skills (STS) scale. The OSATS has accepted construct validity for direct observation ratings by experts to discriminate between trainees' levels of experience. Expert time is scarce. Endoscopic video recordings would facilitate assessment with the OSATS. We aimed to compare video OSATS with direct OSATS. Methods: We included 79 participants with different levels of experience [58 medical students, 15 junior residents (novices), and 6 experts]. Performance of a cadaveric porcine laparoscopic cholecystectomy (LC) was evaluated with OSATS by blinded expert raters by direct observation and then as an endoscopic video recording. Operative time was recorded. Results: Direct OSATS rating and video OSATS rating correlated significantly (ρ = 0.33, p = 0.005). Significant construct validity was found for direct OSATS in distinguishing between students or novices and experts. Students and novices were not different in direct OSATS or video OSATS. Mean operative times varied for students (73.4 ± 9.0 min), novices (65.2 ± 22.3 min), and experts (46.8 ± 19.9 min). Internal consistency was high between the GRS and STS subscores for both direct and video OSATS with Cronbach's α of 0.76 and 0.86, respectively. Video OSATS and operative time in combination was a better predictor of direct OSATS than each single parameter. Conclusion: Direct OSATS rating was better than endoscopic video rating for differentiating between students or novices and experts for LC and should remain the standard approach for the discrimination of experience levels. However, in the absence of experts for direct rating, video OSATS supplemented with operative time should be used instead of single parameters for predicting direct OSATS scores.

show abstract

“…Objective assessment of technical skills of surgical trainees is an important topic in the field of surgical education. Objective structured assessment of technical skills (OSATS) has been implemented to measure MD surgeons' performances for surgical skill assessment . Construct validity of OSATS has been demonstrated using benchtop models, simulation motion metrics, animal models, and operating room performance among MD surgeons .…”

Section: Introductionmentioning

confidence: 99%

“…Objective structured assessment of technical skills (OSATS) has been implemented to measure MD surgeons' performances for surgical skill assessment. 13,[31][32][33][34] Construct validity of OSATS has been demonstrated using benchtop models, simulation motion metrics, animal models, and operating room performance among MD surgeons. 33,[35][36][37][38] A previous study has investigated and in part validated a modified OSATS system for veterinary surgeons, 26 which demonstrated similar result as in MD surgeons.…”

Section: Introductionmentioning

confidence: 99%

Comparison of 2 training programs for basic laparoscopic skills and simulated surgery performance in veterinary students

et al. 2017

View full text Add to dashboard Cite

Objective: To compare the effects of 2 training curricula on laparoscopic skills and performance of simulated surgery in veterinary students.Study design: Prospective study.Sample population: Veterinary students (n 5 33) with no prior hands-on experience in minimally invasive surgery.Methods: Basic laparoscopic skills (BLS) were assessed based on 5 modified McGill inanimate system for training and evaluation of laparoscopic skills. Motion metrics and an objective structured assessment of technical skills (OSATS) were used to evaluate surgical skills during a simulated laparoscopic cholecystectomy performed in an augmented reality simulator. Students were randomly assigned to either skill-based (group A) or procedural-based (group B) training curriculum. Both tests were performed prior to and after a 10-session training curriculum.Results: Post-training BLS results were improved in both training groups (P < .001). Seven participants completed both presimulated and postsimulated laparoscopic cholecystectomy, preventing paired analysis. Based on motion metrics analysis, participants completed tasks in a shorter time (P 5 .0187), and with better economy of movement (P 5 .0457) after training. No difference was detected in OSATS before and after training.Conclusion: Both training curricula improved BLS, but significant differences were not detected between the procedural-based training program and basic skills training alone in veterinary students. Motion metrics such as time, economy of movement, and instrument path were superior to an OSATS, when assessing surgical performance. Further studies are needed to compare the effects of different simulators on the training of veterinarians with diverse laparoscopic surgical experience. | I NT ROD UCTI ONMinimal invasive surgery (MIS) results in less pain and faster recovery for animal patients 1-5 but requires different surgical skills than traditional open surgery. 6,7 Surgeons need to overcome the fulcrum effect, adjust hand-eye coordination, altering the approaching angle of instruments, and adjust to a more restricted field of view. 8 The surgical mantra "see one, do one, teach one" is not suitable in laparoscopic surgery. 9 In the past 2 decades, interest in simulation training and assessment has grown, to surmount the training obstacle in MIS, in animals and man. [8][9][10][11][12][13][14] Simulation aims to replicate clinical situations, ranging from low fidelity simulated basic skills training, to very realistic surgical simulations in high fidelity models. Accordingly, health educators in human medicine have adopted simulation as a viable educational method to teach and practice a diverse range of clinical skills. Live animal or cadaver models are high fidelity and useful training and assessment tools, 13,28 but are sparsely available, very expensive, and the ethical concern of training in live animals remains an important aspect. A recent study showed the value of motion metrics in evaluation of laparoscopic skills in veterinary medicine, 29 yet the effe...

show abstract

Validity and Reliability of Global Operative Assessment of Laparoscopic Skills (GOALS) in Novice Trainees Performing a Laparoscopic Cholecystectomy

Cited by 56 publications

References 21 publications

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Direct Observation versus Endoscopic Video Recording-Based Rating with the Objective Structured Assessment of Technical Skills for Training of Laparoscopic Cholecystectomy

Comparison of 2 training programs for basic laparoscopic skills and simulated surgery performance in veterinary students

Contact Info

Product

Resources

About