The intraoperative activity of a surgeon has substantial impact on postoperative outcomes. However, for most surgical procedures, the details of intraoperative surgical actions, which can vary widely, are not well understood. Here we report a machine learning system leveraging a vision transformer and supervised contrastive learning for the decoding of elements of intraoperative surgical activity from videos commonly collected during robotic surgeries. The system accurately identified surgical steps, actions performed by the surgeon, the quality of these actions and the relative contribution of individual video frames to the decoding of the actions. Through extensive testing on data from three different hospitals located in two different continents, we show that the system generalizes across videos, surgeons, hospitals and surgical procedures, and that it can provide information on surgical gestures and skills from unannotated videos. Decoding intraoperative activity via accurate machine learning systems could be used to provide surgeons with feedback on their operating skills, and may allow for the identification of optimal surgical behaviour and for the study of relationships between intraoperative factors and postoperative outcomes.

Purpose:Previously, we identified 8 objective suturing performance metrics highly predictive of urinary continence recovery after robotic-assisted radical prostatectomy. Here, we aimed to test the feasibility of providing tailored feedback based upon these clinically relevant metrics and explore the impact on the acquisition of robotic suturing skills.Materials and Methods:Training surgeons were recruited and randomized to a feedback group or a control group. Both groups completed a baseline, midterm and final dry laboratory vesicourethral anastomosis (VUA) and underwent 4 intervening training sessions each, consisting of 3 suturing exercises. Eight performance metrics were recorded during each exercise: 4 automated performance metrics (derived from kinematic and system events data of the da Vinci® Robotic System) representing efficiency and console manipulation competency, and 4 suturing technical skill scores. The feedback group received tailored feedback (a visual diagram+verbal instructions+video examples) based on these metrics after each session. Generalized linear mixed model was used to compare metric improvement (Δ) from baseline to the midterm and final VUA.Results:Twenty-three participants were randomized to the feedback group (11) or the control group (12). Demographic data and baseline VUA metrics were comparable between groups. The feedback group showed greater improvement than the control group in aggregate suturing scores at midterm (mean Δ feedback group 4.5 vs Δ control group 1.1) and final VUA (Δ feedback group 5.3 vs Δ control group 4.9). The feedback group also showed greater improvement in the majority of the included metrics at midterm and final VUA.Conclusions:Tailored feedback based on specific, clinically relevant performance metrics is feasible and may expedite the acquisition of robotic suturing skills.

Artificial intelligence (AI) systems can now reliably assess surgeon skills through videos of intraoperative surgical activity. With such systems informing future high-stakes decisions such as whether to credential surgeons and grant them the privilege to operate on patients, it is critical that they treat all surgeons fairly. However, it remains an open question whether surgical AI systems exhibit bias against surgeon sub-cohorts, and, if so, whether such bias can be mitigated. Here, we examine and mitigate the bias exhibited by a family of surgical AI systems—SAIS—deployed on videos of robotic surgeries from three geographically-diverse hospitals (USA and EU). We show that SAIS exhibits an underskilling bias, erroneously downgrading surgical performance, and an overskilling bias, erroneously upgrading surgical performance, at different rates across surgeon sub-cohorts. To mitigate such bias, we leverage a strategy —TWIX—which teaches an AI system to provide a visual explanation for its skill assessment that otherwise would have been provided by human experts. We show that whereas baseline strategies inconsistently mitigate algorithmic bias, TWIX can effectively mitigate the underskilling and overskilling bias while simultaneously improving the performance of these AI systems across hospitals. We discovered that these findings carry over to the training environment where we assess medical students’ skills today. Our study is a critical prerequisite to the eventual implementation of AI-augmented global surgeon credentialing programs, ensuring that all surgeons are treated fairly.

Background Surgeons who receive reliable feedback on their performance quickly master the skills necessary for surgery. Such performance-based feedback can be provided by a recently-developed artificial intelligence (AI) system that assesses a surgeon’s skills based on a surgical video while simultaneously highlighting aspects of the video most pertinent to the assessment. However, it remains an open question whether these highlights, or explanations, are equally reliable for all surgeons. Methods Here, we systematically quantify the reliability of AI-based explanations on surgical videos from three hospitals across two continents by comparing them to explanations generated by humans experts. To improve the reliability of AI-based explanations, we propose the strategy of training with explanations –TWIX –which uses human explanations as supervision to explicitly teach an AI system to highlight important video frames. Results We show that while AI-based explanations often align with human explanations, they are not equally reliable for different sub-cohorts of surgeons (e.g., novices vs. experts), a phenomenon we refer to as an explanation bias. We also show that TWIX enhances the reliability of AI-based explanations, mitigates the explanation bias, and improves the performance of AI systems across hospitals. These findings extend to a training environment where medical students can be provided with feedback today. Conclusions Our study informs the impending implementation of AI-augmented surgical training and surgeon credentialing programs, and contributes to the safe and fair democratization of surgery.

No abstract