Novel evaluation of surgical activity recognition models using task-based efficiency metrics

Abstract

Purpose: Surgical task-based metrics (rather than entire procedure metrics) can be used to improve surgeon training and, ultimately, patient care through focused training interventions. Machine learning models to automatically recognize individual tasks or activities are needed to overcome the otherwise manual effort of video review. Traditionally, these models have been evaluated using frame-level accuracy. Here, we propose evaluating surgical activity recognition models by their effect on task-based efficiency metrics. In this way, we can determine when models have achieved adequate performance for providing surgeon feedback via metrics from individual tasks. Methods: We propose a new CNN-LSTM model, RP-Net-V2, to recognize the 12 steps of robotic-assisted radical prostatectomies (RARP). We evaluated our model both in terms of conventional methods (e.g. Jaccard Index, task boundary accuracy) as well as novel ways, such as the accuracy of efficiency metrics computed from instrument movements and system events.Results: Our proposed model achieves a Jaccard Index of 0.85 thereby outperforming previous models on robotic-assisted radical prostatectomies. Additionally, we show that metrics computed from tasks automatically identified using RP-Net-V2 correlate well with metrics from tasks labeled by clinical experts. Conclusions: We demonstrate that metrics-based evaluation of surgical activity recognition models is a viable approach to determine when models can be used to quantify surgical efficiencies. We believe this approach and our results illustrate the potential for fully automated, post-operative efficiency reports.

Evolving robotic surgery training and improving patient safety, with the integration of novel technologies

“…If these surgical activity recognition models could assess each phase of a surgical procedure, they could be utilised to compute efficiency metrics. Nevertheless, most models reported in the literature ranged from around 50–80% accuracy [ 44 ]. Recognition of anatomical landmarks is a current limitation of APMs evaluating surgical skills.…”

Automatic generation of checklists from business process model and notation (BPMN) models for surgical assist systems

“…Workflow supporting systems are the research field of several workgroups, but most of them focus on automatic situation recognition to determine the current state of the surgery [9][10][11]. In [12] a "surgical procedure manager" is used to guide the surgeon through an endonasal procedure using pre-defined checklist items, which are confirmed by a footswitch.…”

The role of AI technology in prediction, diagnosis and treatment of colorectal cancer

“…An initial SVM classifier system has been presented with remarkable accuracy in the assessment of surgical performance or skills, even is fed by limited data (Ghani et al 2016 ; Hung et al 2018c ). Increasing AI algorithms and models have been developed (Table 3 ) (Hung et al 2019 ; Zia et al 2018 , 2019 ; Schroerlucke et al 2017 ; Rao et al 2017 ; Guan et al 2017 ). We believe video-based surgery, both robotic and laparoscopy, is a fertile soil for AI to land on.…”