SATR-DL: Improving Surgical Skill Assessment And Task Recognition In Robot-Assisted Surgery With Deep Neural Networks

Abstract

Purpose: This paper focuses on an automated analysis of surgical motion profiles for objective skill assessment and task recognition in robot-assisted surgery. Existing techniques heavily rely on conventional statistic measures or shallow modelings based on hand-engineered features and gesture segmentation. Such developments require significant expert knowledge, are prone to errors, and are less efficient in online adaptive training systems. Methods: In this work, we present an efficient analytic framework with a parallel deep learning architecture, SATR-DL, to assess trainee expertise and recognize surgical training activity. Through an end-to-end learning technique, abstract information of spatial representations and temporal dynamics is jointly obtained directly from raw motion sequences. Results: By leveraging a shared highlevel representation learning, the resulting model is successful in the recognition of trainee skills and surgical tasks, suturing, needle-passing, and knot-tying. Meanwhile, we explore the use of ensemble in classification at the trial level, where the SATR-DL outperforms state-of-the-art performance by achieving accuracies of 0.960 and 1.000 in skill assessment and task recognition, respectively. Conclusion: This study highlights the potential of SATR-DL to provide improvements for an efficient data-driven assessment in intelligent robotic surgery.

A Method for Segmenting the Process of Needle Insertion during Simulated Cannulation using Sensor Data

“…It can expose areas of weakness in participants' technique or skill and provide objective metrics for assessment and find areas of improvement. As machine learning and neural network algorithms start to show great promise in clinical training and assessment [16], [17], the algorithm proposed here can be improved to analyze large groups of data to sort performance based on skill levels. Assessing data in this way could provide insight into improvements that clinicians could make for more efficient and safe cannulation.…”

A Review on Deep Learning in Minimally Invasive Surgery

“…A similar approach is presented in [83], which model is able to reliably interpret skills within a 1-3 second window, without needing an observation of the entire training trial. Wang et al [84] proposed a multi-output model, SATR-DL, for online trainee skill analysis and task recognition, achieving accuracies of 96% and 100% for these two tasks.…”

Applications of Haptic Technology, Virtual Reality, and Artificial Intelligence in Medical Training During the COVID-19 Pandemic

“… Wang et al (2018) suggests that conventional CNN falls short compared to traditional hand-crafted feature extraction techniques because it only considers the appearances (spatial features) and ignores the data’s temporal dynamics. In Wang and Fey (2018) , a parallel deep learning architecture is proposed to recognize the surgical training activity and assess trainee expertise. A Gated recurrent unit (GRU) is used for temporal feature extraction, and a CNN network is used to extract the spatial features.…”