Deep Learning to Automate Technical Skills Assessment in Robotic Surgery

Abstract

Surgeon performance affects patient outcomes. 1 To improve patient outcomes, we must identify poor surgical performance. However, surgeons may not always associate a specific surgical act with its consequential outcome unless the error is egregious and the outcome is immediate. Today, there is little formal structure for surgeons to receive specific technical skills feedback after formal training. Current hurdles for surgeons to obtain and maintain hospital privileges to perform an operative procedure include peer proctoring and evaluation, which are arguably insufficient when juxtaposed to the potentially devastating outcomes that can occur if surgical errors arise.We have proposed automating technical skills assessment in robotic surgery using deep learning, a subdiscipline of artificial intelligence that uses neural networklike algorithms to learn from data sets and make complex decisions. The emphasis of our work is on actionable real-time feedback. Ideally, this automated assessment only requires video footage of the surgeon's performance, whereby computer vision, harnessing deep learning algorithms, can learn from past master surgeon movements and then prospectively and objectively identify specific areas where a new learning surgeon can improve (Figure).

Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications

“…Throughout surgical data science, researchers are interested in exploring the associations among surgeon characteristics, tool movement, task outcome, and, eventually, patient outcome, complication occurrence, and technical competency. 44 Surgical video is the primary data source for this information for most operations aside from a few robotic-assisted procedures. 45 , 46 , 47 Despite its importance, no publicly available data sets exist of surgical video labeled with clinically meaningful outcomes.…”

Tailored Feedback Based on Clinically Relevant Performance Metrics Expedites the Acquisition of Robotic Suturing Skills—An Unblinded Pilot Randomized Controlled Trial

“…24,25 On this aspect, we have used a deeplearning algorithm to automate suturing technical skill assessment in the virtual-reality environment and achieved promising preliminary performance. 26 Though the present study identified feasible and effective feedback based on clinically relevant outcomes to improve surgeon performance, there are some limitations. First, this was a pilot study with a limited sample size.…”

“…24,25 On this aspect, we have used a deep-learning algorithm to automate suturing technical skill assessment in the virtual-reality environment and achieved promising preliminary performance. 26…”

“…In our preliminary studies, we showed that APMs, specifically those in the vesicourethral anastomosis (VUA) step, together with suturing technical skills (ie RACE scores), were strong predictors of continence recovery after RARP. 3,13,14 In this pilot study, we aimed to test the concept that tailored feedback, based upon these clinically relevant performance metrics, can expedite the acquisition of robotic suturing skills in a dry laboratory environment.…”

“…7 Additionally, automated performance metrics (APMs), derived from instrument kinematic and system events data during robotic surgery, have been shown to predict urinary continence recovery after RARP better than surgeon experience. 3 Although these clinically relevant assessments have been extensively validated in the literature, few studies have utilized them to provide feedback for surgical training. Traditionally, surgical skills were taught using an apprenticeship model where training surgeons would observe and learn from a mentor, and eventually participate in the case.…”