A vision transformer for decoding surgeon activity from surgical videos

Abstract

The intraoperative activity of a surgeon has substantial impact on postoperative outcomes. However, for most surgical procedures, the details of intraoperative surgical actions, which can vary widely, are not well understood. Here we report a machine learning system leveraging a vision transformer and supervised contrastive learning for the decoding of elements of intraoperative surgical activity from videos commonly collected during robotic surgeries. The system accurately identified surgical steps, actions performed by the surgeon, the quality of these actions and the relative contribution of individual video frames to the decoding of the actions. Through extensive testing on data from three different hospitals located in two different continents, we show that the system generalizes across videos, surgeons, hospitals and surgical procedures, and that it can provide information on surgical gestures and skills from unannotated videos. Decoding intraoperative activity via accurate machine learning systems could be used to provide surgeons with feedback on their operating skills, and may allow for the identification of optimal surgical behaviour and for the study of relationships between intraoperative factors and postoperative outcomes.

A multi-institutional study using artificial intelligence to provide reliable and fair feedback to surgeons

“…After completing their training process, raters were asked to annotate the video samples in this study with binary skill assessments (low vs. high skill). In the event of disagreements in the annotations, we followed the same strategy adopted in the original study 10 where the lowest of all scores is considered as the final annotation.…”

“…SAIS is an AI system for skill assessment. We recently developed SAIS to decode the intraoperative activity of surgeons based exclusively on surgical videos 10 . Specifically, we demonstrated state-of-the-art performance in assessing the skill-level of surgical activity, such as needle handling and needle driving, across multiple hospitals.…”

“…To that end, we recently developed a surgical artificial intelligence system (SAIS) that assesses the skill of a surgeon based on a video of intraoperative activity and simultaneously highlights video frames of pertinent activity. We demonstrated that SAIS reliably automates surgeon skill assessment 10 and also examined the fairness of its assessments 11 . In pursuit of developing a trustworthy AI system capable of safely automating the provision of feedback to surgeons, we must ensure that SAIS' video-frame highlights, which we refer to as AI-based explanations, align with the expectations of expert surgeons (i.e., be reliable) 12,13 and be equally reliable for all surgeons (i.e., be fair) 14 .…”

“…An element with a higher attention score is assumed to be more important than that with a lower score. Similarly, with a transformer architecture which operates on videos, SAIS 10 generates an attention score for each frame in a surgical video, with high-attention frames assumed to be more relevant to the surgeon skill assessment. When such an AI system's explanations align with those provided by humans, it can direct surgeons to specific aspects of their operating technique that require improvement while simultaneously enhancing its trustworthiness 19 .…”

“…In this study, we quantify the reliability and fairness of explanations generated by a surgical AI system -SAIS 10 -that we previously developed and which was shown to reliably assess the skill level of surgeons from videos. Through evaluation on data from three geographically-diverse hospitals, we show that SAIS' attention-based explanations often align, albeit imperfectly, with human explanations.…”