Temporal clustering of surgical activities in robot-assisted surgery

Abstract

PurposeMost evaluations of surgical workflow or surgeon skill use simple, descriptive statistics (e.g., time) across whole procedures, thereby deemphasizing critical steps and potentially obscuring critical inefficiencies or skill deficiencies. In this work, we examine off-line, temporal clustering methods that chunk training procedures into clinically relevant surgical tasks or steps during robot-assisted surgery.MethodsWe collected system kinematics and events data from nine surgeons performing five different surgical tasks on a porcine model using the da Vinci Si surgical system. The five tasks were treated as one ‘pseudo-procedure.’ We compared four different temporal clustering algorithms—hierarchical aligned cluster analysis (HACA), aligned cluster analysis (ACA), spectral clustering (SC), and Gaussian mixture model (GMM)—using multiple feature sets.ResultsHACA outperformed the other methods reaching an average segmentation accuracy of when using all system kinematics and events data as features. SC and ACA reached moderate performance with and average segmentation accuracy, respectively. GMM consistently performed poorest across algorithms.ConclusionsUnsupervised temporal segmentation of surgical procedures into clinically relevant steps achieves good accuracy using just system data. Such methods will enable surgeons to receive directed feedback on individual surgical tasks rather than whole procedures in order to improve workflow, assessment, and training.

Unsupervised Learning for Surgical Motion by Learning to Predict the Future

“…The most relevant prior work is [9], which encodes short windows of kinematics using denoising autoencoders, and which uses these representations to search a database using motion-based queries. Other unsupervised approaches include activity alignment under the assumption of identical structure [10] and activity segmentation using hand-crafted pipelines [6], structured probablistic models [14], and clustering [20].…”

Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

“…The recognition and segmentation of the robot's current action is one of the main pillars of the surgical state estimation process. Many models have been developed for the segmentation and recognition of finegrained surgical actions that last for a few seconds, such as cutting [5][6][7][8], as well as surgical phases that last for up to 10 minutes, such as bladder dissection [9][10][11]. The recognition of fine-grained surgical states is particularly challenging due to their short duration and frequent state transitions.…”

“…Lea et al measured two scene-based features in JIGSAWS as additional variables to the robot kinematics data in their Latent Convolutional Skip-Chain CRF (LC-SC-CRF) model [5]. Zia et al collected the robot kinematics and system events data from RAS to perform surgical phase recognition [10]. While these attempts have proven to improve the model accuracy, to the best of the authors' knowledge, there is yet to be a unified method that incorporates multiple data sources directly for fine-grained surgical state estimation.…”

Novel evaluation of surgical activity recognition models using task-based efficiency metrics

“…The begin and end times of tasks or sub-tasks must be automatically identified from within the entire procedure because manual identification through post-operative video review is overly time consuming and not scalable. Machine learning algorithms have been used with promising initial results in laparoscopic [8,9,10,11] and robotic-assisted surgeries [12,13,14,15,16,17,18].…”