Use of surgical video–based automated performance metrics to predict blood loss and success of simulated vascular injury control in neurosurgery: a pilot study

Abstract

OBJECTIVE Experts can assess surgeon skill using surgical video, but a limited number of expert surgeons are available. Automated performance metrics (APMs) are a promising alternative but have not been created from operative videos in neurosurgery to date. The authors aimed to evaluate whether video-based APMs can predict task success and blood loss during endonasal endoscopic surgery in a validated cadaveric simulator of vascular injury of the internal carotid artery. METHODS Videos of cadaveric simulation trials by 73 neurosurgeons and otorhinolaryngologists were analyzed and manually annotated with bounding boxes to identify the surgical instruments in the frame. APMs in five domains were defined—instrument usage, time-to-phase, instrument disappearance, instrument movement, and instrument interactions—on the basis of expert analysis and task-specific surgical progressions. Bounding-box data of instrument position were then used to generate APMs for each trial. Multivariate linear regression was used to test for the associations between APMs and blood loss and task success (hemorrhage control in less than 5 minutes). The APMs of 93 successful trials were compared with the APMs of 49 unsuccessful trials. RESULTS In total, 29,151 frames of surgical video were annotated. Successful simulation trials had superior APMs in each domain, including proportionately more time spent with the key instruments in view (p < 0.001) and less time without hemorrhage control (p = 0.002). APMs in all domains improved in subsequent trials after the participants received personalized expert instruction. Attending surgeons had superior instrument usage, time-to-phase, and instrument disappearance metrics compared with resident surgeons (p < 0.01). APMs predicted surgeon performance better than surgeon training level or prior experience. A regression model that included APMs predicted blood loss with an R2 value of 0.87 (p < 0.001). CONCLUSIONS Video-based APMs were superior predictors of simulation trial success and blood loss than surgeon characteristics such as case volume and attending status. Surgeon educators can use APMs to assess competency, quantify performance, and provide actionable, structured feedback in order to improve patient outcomes. Validation of APMs provides a benchmark for further development of fully automated video assessment pipelines that utilize machine learning and computer vision.

Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications

“…[1][2][3][4][5] Applying these methods to analyze surgical video from the operating room may improve patient care by detecting complications, assisting trainee development, and assessing performance. [6][7][8][9][10] Algorithmic methods thrive when presented with comprehensive data sets. Surgical video data sets, however, are traditionally collected to answer a single question by a single group of investigators, depicting a particular surgical scenario (eg, one data set for object detection in neurosurgery and a different data set for phase detection in general surgery).…”

“…Advances in machine learning and computer vision (CV) facilitate the discovery of unknown patterns in large bodies of visual data . Applying these methods to analyze surgical video from the operating room may improve patient care by detecting complications, assisting trainee development, and assessing performance . Algorithmic methods thrive when presented with comprehensive data sets.…”

Evaluation of a scoring system to assess proficiency in cerebral angiography for neuroendovascular surgery education

“…The lowest average score among the ten items was for catheter operation, which is a unique technique that cannot be obtained through the performance of open surgery. It has already been reported that physicians perform better surgery when there is smaller blood loss, and it is considered that the same applies to angiography and endovascular treatment [ 10 ]. The item procedural completion assesses whether the physician needed to be replaced by a senior physician because of dangerous or unsuccessful repeated procedures.…”

The future of artificial intelligence in neurosurgery: A narrative review

“…The APMs were found to be more accurate predictors of surgeons’ performance than their training status or prior experience. [ 33 ] With the rapidly advancing AI technologies, robots requiring minimal human supervision have led to reduced surgical site infections, precise tumor segmentation, reduced incision, and scar size and shorter recovery spans. [ 32 ] Although these ML powered devices can have unprecedented potential of pairing with human teammates, they do not necessarily possess human teamwork competencies such as communication, coordination, and adaptation.…”