Crowd-Sourced Assessment of Technical Skills for Validation of Basic Laparoscopic Urologic Skills Tasks

Kowalewski, Timothy M.; Comstock, Bryan A.; Sweet, Robert; Schaffhausen, Cory R.; Menhadji, Ashleigh; Averch, Timothy D.; Box, Geoffrey N.; Brand, Timothy C.; Ferrandino, Michael N.; Kaouk, Jihad; Knudsen, Bodo E.; Landman, Jaime; Lee, Benjamin; Schwartz, Bradley F.; McDougall, Elspeth M.; Lendvay, Thomas S.

doi:10.1016/j.juro.2016.01.005

Cited by 48 publications

(29 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study used the Basic Laparascopic Urologic Study (BLUS) dataset, described in detail in [1]. This dataset arose from a gap in the field, in which no educational surgical certification process existed for urologic surgery, as opposed to how the Fundamentals of Laparosocopic Surgery (FLS) exists for general surgical procedures [19], [20], [21].…”

Section: Datasetmentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

See 1 more Smart Citation

The effect of video playback speed on surgeon technical skill perception

et al. 2020

Self Cite

View full text Add to dashboard Cite

Purpose: Finding effective methods of discriminating surgeon technical skill has proven a complex problem to solve computationally. Previous research has shown that obtaining non-expert crowd evaluations of surgical performances is as accurate as the gold standard, expert surgeon review [1]. The aim of this research is to learn whether crowdsourced evaluators give higher ratings of technical skill to video of performances with increased playback speed, its effect in discriminating skill levels, and whether this increase is related to the evaluator consciously being aware that the video is being manually manipulated. Methods: A set of ten peg transfer videos (5 novices, 5 experts), were used to evaluate the perceived technical skill of the performers at each video playback speed used (0.4x-3.6x). Objective metrics used for measuring technical skill were also computed for comparison by manipulating the corresponding kinematic data of each performance. Two videos of an expert and novice performing dry lab laparoscopic trials of peg transfer tasks were used to obtain evaluations at each playback speed (0.2x-3.0x) of perception of whether a video is played at real-time playback speed or not. Results: We found that while both novices and experts had increased perceived technical skill as the video playback was increased, the amount of increase was significantly greater for experts. Each increase in the playback speed by 0.4x was associated with, on average, a 0.72-point increase in the GOALS score (95% CI: 0.60-0.84 point increase; p < 0.001) for expert videos and only a 0.24-point increase in the GOALS score (95% CI: 0.13-0.36 point increase; p < 0.001) for novice videos. Conclusion: Due to the differential increase in perceived technical skill due to increased playback speed for experts, the difference between novice and expert skill levels of surgi-

show abstract

Section: Datasetmentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

The effect of video playback speed on surgeon technical skill perception

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…[3][4][5] Recently, evaluation of video footage by laypeople trained in the use of validated assessment metrics, termed "crowdsourcing," has been shown to provide efficient and reliable feedback that correlates well with expert ratings. 6,7 This methodology has been translated to differentiating surgeon skill 8 in robotic prostatectomy. In order to improve care provided to patients, one aspect of surgical quality improvement seeks to identify both intraoperative steps that contribute to clinically relevant outcomes.…”

Section: Introductionmentioning

confidence: 99%

Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy

Goldenberg

Nabhani

Wallis

et al. 2017

CUAJ

View full text Add to dashboard Cite

Introduction: Development of uretero-ileal stricture (UIS) after robotic-assisted radical cystectomy (RARC) may be dependent on surgical technique. Video review of intraoperative technique is an emerging paradigm for surgical quality improvement. We examined whether surgeon-perceived risk of UIS or crowd-sourced assessment of robotic skill are associated with the development of UIS. Methods: We conducted a case-control study comparing the operative technique of uretero-ileal anastomoses resulting in clinically significant UIS with the contralateral anastomosis for the same patient. De-identified videos were analyzed by 1) five high-volume surgeons; and 2) crowd workers (Crowd-Sourced Assessment of Technical Skill, C-SATS) to determine Global Evaluative Assessment of Robotic Skill (GEARS) score. Mantel-Haenszel common odds ratio (OR) estimates were calculated to assess the association between surgeon performance and the development of UIS. Logistic regression models were used to examine the association between GEARS scores and the development of UIS. Results: A total of 10 UIS videos were compared to eight control videos by five surgeons and 2142 crowd workers. Expert surgeons systematically evaluated intraoperative footage, however, no association between the expert mode response and UIS (OR 0.42; 95% confidence interval [CI] 0.05-3.45; p=0.91) was identified. Crowd-sourced assessment was not predictive of UIS (p=0.62). Conclusions: We used video review to systematically analyze procedure-specific content and technique. The inability of surgeons to predict UIS may reflect the questionnaire, uncontrolled patient factors, or a lack of power. Crowd-sourced GEARS score was unsuccessful in predicting UIS after RARC.

show abstract

“…The authors compared assessments by expert raters (robotic and open surgeons) and those from lay people from the Crowd-Sourced Assessment of Technical Skills (C-SATS) 1,2 group with regard to experienced surgeons' technical ability during a robotic-assisted radical cystectomy. The authors prepared short video segments (60 seconds) showing mobilization of the ureter, ureteral preparation for the anastomosis, and the ureteral-ileal anastomosis from nine (out of a potential 102) cases that resulted in clinically significant postoperative uretero-ileal strictures (UIS) (10 strictures in total).…”

mentioning

confidence: 99%

Video replay in surgery: Can we make the “right call” in predicting outcomes?

Matsumoto

2017

CUAJ

View full text Add to dashboard Cite

Crowd-Sourced Assessment of Technical Skills for Validation of Basic Laparoscopic Urologic Skills Tasks

Cited by 48 publications

References 19 publications

The effect of video playback speed on surgeon technical skill perception

The effect of video playback speed on surgeon technical skill perception

Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy

Video replay in surgery: Can we make the “right call” in predicting outcomes?

Contact Info

Product

Resources

About