Weakly Supervised Recognition of Surgical Gestures

Abstract

Kinematic trajectories recorded from surgical robots contain information about surgical gestures and potentially encode cues about surgeon's skill levels. Automatic segmentation of these trajectories into meaningful action units could help to develop new metrics for surgical skill assessment as well as to simplify surgical automation. State-of-the-art methods for action recognition relied on manual labelling of large datasets, which is time consuming and error prone. Unsupervised methods have been developed to overcome these limitations. However, they often rely on tedious parameter tuning and perform less well than supervised approaches, especially on data with high variability such as surgical trajectories. Hence, the potential of weak supervision could be to improve unsupervised learning while avoiding manual annotation of large datasets. In this paper, we used at a minimum one expert demonstration and its ground truth annotations to generate an appropriate initialization for a GMM-based algorithm for gesture recognition. We showed on real surgical demonstrations that the latter significantly outperforms standard task-agnostic initialization methods. We also demonstrated how to improve the recognition accuracy further by redefining the actions and optimising the inputs.

Gesture Recognition in Robotic Surgery: A Review

“…While unsupervised approaches mitigate significantly labelling costs and issues, as labels are only necessary for model testing, they still show a significant gap in performance with respect to the supervised methods. A midway solution is represented by semi-supervised learning [73,74,75,76], where a small pool of annotated demonstrations is used for model training. Such demonstrations can be used to transfer gesture labels to a large set of unlabelled observations previously aligned with DTW [73], or to initialize clustering of unlabelled data and avoid tedious parameter tuning [74].…”

“…A midway solution is represented by semi-supervised learning [73,74,75,76], where a small pool of annotated demonstrations is used for model training. Such demonstrations can be used to transfer gesture labels to a large set of unlabelled observations previously aligned with DTW [73], or to initialize clustering of unlabelled data and avoid tedious parameter tuning [74]. These systems are however highly sensitive to large data variability, showing rapid drop in recognition accuracy with increased heterogeneity in terms of gesture ordering or surgical skill level respectively.…”

Computer Vision in the Surgical Operating Room

“…The JIG-SAWS dataset has extended annotations at a sub-task level. AI techniques learn patterns and temporal interconnections of the sub-task sequences from combinations of robot kinematics and surgical video and detect and temporally localize each sub-task [19][20][21][22][23][24]. Recently, AI models for activity recognition have been developed and tested on annotated datasets from real cases of robotic-assisted radical prostatectomy and ocular microsurgery [18][19][20].…”

Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

“…Zappella et al proposed methods of modeling surgical video clips for single-action classification [16]. The Transition State Clustering (TSC) and Gaussian Mixture Model methods provide unsupervised or weakly-supervised methods for surgical trajectory segmentation [17,18]. More recently, deep learning methods have come to define the state-of-the-art, such as Temporal Convolutional Networks (TCN) [19], Time Delay Neural Network (TDNN) [7], and Long-Short Term Memory (LSTM) [6,20].…”