Expert surgeons and deep learning models can predict the outcome of surgical hemorrhage from 1 min of video

Abstract

Major vascular injury resulting in uncontrolled bleeding is a catastrophic and often fatal complication of minimally invasive surgery. At the outset of these events, surgeons do not know how much blood will be lost or whether they will successfully control the hemorrhage (achieve hemostasis). We evaluate the ability of a deep learning neural network (DNN) to predict hemostasis control ability using the first minute of surgical video and compare model performance with human experts viewing the same video. The publicly available SOCAL dataset contains 147 videos of attending and resident surgeons managing hemorrhage in a validated, high-fidelity cadaveric simulator. Videos are labeled with outcome and blood loss (mL). The first minute of 20 videos was shown to four, blinded, fellowship trained skull-base neurosurgery instructors, and to SOCALNet (a DNN trained on SOCAL videos). SOCALNet architecture included a convolutional network (ResNet) identifying spatial features and a recurrent network identifying temporal features (LSTM). Experts independently assessed surgeon skill, predicted outcome and blood loss (mL). Outcome and blood loss predictions were compared with SOCALNet. Expert inter-rater reliability was 0.95. Experts correctly predicted 14/20 trials (Sensitivity: 82%, Specificity: 55%, Positive Predictive Value (PPV): 69%, Negative Predictive Value (NPV): 71%). SOCALNet correctly predicted 17/20 trials (Sensitivity 100%, Specificity 66%, PPV 79%, NPV 100%) and correctly identified all successful attempts. Expert predictions of the highest and lowest skill surgeons and expert predictions reported with maximum confidence were more accurate. Experts systematically underestimated blood loss (mean error − 131 mL, RMSE 350 mL, R2 0.70) and fewer than half of expert predictions identified blood loss > 500 mL (47.5%, 19/40). SOCALNet had superior performance (mean error − 57 mL, RMSE 295 mL, R2 0.74) and detected most episodes of blood loss > 500 mL (80%, 8/10). In validation experiments, SOCALNet evaluation of a critical on-screen surgical maneuver and high/low-skill composite videos were concordant with expert evaluation. Using only the first minute of video, experts and SOCALNet can predict outcome and blood loss during surgical hemorrhage. Experts systematically underestimated blood loss, and SOCALNet had no false negatives. DNNs can provide accurate, meaningful assessments of surgical video. We call for the creation of datasets of surgical adverse events for quality improvement research.

Automated Capture of Intraoperative Adverse Events Using Artificial Intelligence: A Systematic Review and Meta-Analysis

“…A flow-chart of the article selection process is shown in Figure 1 . The 13 studies included in this review were published from 2016 to 2022 [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ], with the majority published during or after 2020. Table 1 shows an overview of the methodologies used, adverse events analyzed, AI algorithms, type of validation, outcomes, and comparative metrics from the 13 included articles.…”

Pilot Analysis of Surgeon Instrument Utilization Signatures Based on Shannon Entropy and Deep Learning for Surgeon Performance Assessment in a Cadaveric Carotid Artery Injury Control Simulation

“…To overcome the current dependence of surgical performance evaluation on time-consuming manual review, automated review of surgical video using clinically derived metrics indicative of surgical skill but generated automatically using DL has been pursued. 8,13,15,16,18,32,33…”

“…15 When interpreting CV-generated tools detections, the regression model also had similar predictive accuracy as expert observers watching 1 minute of the trial video (73.9% using CV detections compared with 70% by experts). 16 Importantly, in our experiment, ShEn for testing set trials was calculated using both DL-generated instrument detections as well as ground truth annotations. We demonstrate that ShEn is robust to detection errors by showing comparable AP between manual, human-identified ground truth tool annotations and tools detected by automated DL ( Supplemental Figure 3 , http://links.lww.com/ONS/A955 and Supplemental Figure 4 , http://links.lww.com/ONS/A956).…”

“…To overcome the current dependence of surgical performance evaluation on timeconsuming manual review, automated review of surgical video using clinically derived metrics indicative of surgical skill but generated automatically using DL has been pursued. 8,13,15,16,18,32,33 Generating meaningful automated performance metrics, or APMs, is critical to their utility as an objective assessment modality for surgical performance. We chose to evaluate surgical instrument usage patterns through a simple mathematical formula, ShEn, that describes the "complexity" of a surgeon's instrument usage.…”

“…13,14 The features of instrument movement can be used to calculate Automated Performance Metrics (APMs), which represent quantitative and summative values based on the surgical episode. 15,16 APMs include metrics calculated from surgeon hand movements, instrument kinematics, and other surgical events. For example, APMs include time to complete a surgical action (such as suturing) or phase (such as tumor resection), first-order, second-order, and thirdorder instrument kinematics, and visual features of the surgical scene (such as obscuration by blood).…”