Automatic data-driven real-time segmentation and recognition of surgical workflow

Dergachyova, Olga; Bouget, David; Huaulmé, Arnaud; Morandi, Xavier; Jannin, Pierre

doi:10.1007/s11548-016-1371-x

Cited by 117 publications

(79 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These transformations allow the execution of more complex procedures and also increase the amount of information available in modern OR. To better tackle this new OR scenario, the context-aware systems (CAS) have gradually been developed to provide detailed comprehension of rich information and contextual support to the clinicians (Bricon-Souf & Newman (2007); Dergachyova et al (2016)). With interpreting the operation procedure and tool usage, automated surgical phase recognition and tool presence detection serve as the primary functions in CAS and such accurate systems are expected to be highly demanded (Padoy et al (2012); Lalys & Jannin (2014); Wesierski & Jezierska (2018)).…”

Section: Introductionmentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

161

View full text Add to dashboard Cite

Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis and also very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both tasks. Specifically, our proposed MTRCNet-CL model has an end-to-end architecture with two branches, which share earlier feature encoders to extract general visual features while holding respective higher layers targeting for specific tasks. Given that temporal information is crucial for phase recognition, long-short term memory (LSTM) is explored to model the sequential dependencies in the phase recognition branch. More importantly, a novel and effective correlation loss is designed to model the relatedness between tool presence and phase identification of each video frame, by minimizing the divergence of predictions from the two branches. Mutually leveraging both lowlevel feature sharing and high-level prediction correlating, our MTRCNet-CL method can encourage the interactions between the two tasks to a large extent, and hence can bring about benefits to each other. Extensive experiments on a large surgical video dataset (Cholec80) demonstrate outstanding performance of our proposed method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 89.1% v.s. 81.0% for the mAP in tool presence detection and 87.4% v.s. 84.5% for F1 score in phase recognition). The code can be found on our project website.

show abstract

Section: Introductionmentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

161

View full text Add to dashboard Cite

show abstract

“…Recently, Forestier et al [28] proposed a method designed to create an online one-dimensional alignment. The second aspect pertains to creating a reliable and automatic online activity recognition method [29,30,31]. With these two developments available, a real-time implementation of our method will be rendered possible.…”

Section: Resultsmentioning

confidence: 99%

Offline identification of surgical deviations in laparoscopic rectopexy

Huaulmé

Jannin

Reche

et al. 2020

Artificial Intelligence in Medicine

Self Cite

View full text Add to dashboard Cite

Objective: A median of 14.4% of patient undergone at least one adverse event during surgery and a third of them are preventable. The occurrence of adverse events forces surgeons to implement corrective strategies and, thus, deviate from the standard surgical process. Therefore, it is clear that the automatic identification of adverse events is a major challenge for patient safety. In this paper, we have proposed a method enabling us to identify such deviations. We have focused on identifying surgeons' deviations from standard surgical processes due to surgical events rather than anatomic specificities. This is particularly challenging, given the high variability in typical surgical procedure workflows. Methods:We have introduced a new approach designed to automatically detect and distinguish surgical process deviations based on multi-dimensional non-linear temporal scaling with a hidden semi-Markov model using manual annotation of surgical processes. The approach was then evaluated using crossvalidation.Results: The best results have over 90 % accuracy. Recall and precision were superior at 70 %. We have provided a detailed analysis of the incorrectlydetected observations.Conclusion: Multi-dimensional non-linear temporal scaling with a hidden semi-Markov model provides promising results for detecting deviations. Our error analysis of the incorrectly-detected observations offers different leads in order to further improve our method.Significance: Our method demonstrated the feasibility of automatically detecting surgical deviations that could be implemented for both skill analysis and developing situation awareness-based computer-assisted surgical systems.

show abstract

“…DTW algorithm produces the best performance detection accuracy 76.8%. Dergachyova et al [30] based on the dataset of laparoscopic cholecystectomy [2] combined surgical instrument data to detect surgical procedures. This method firstly models the surgical process, performs feature extraction on visual and surgical instruments, classifies the features using AdaBoost, and finally generates the decision using a hidden Markov model.…”

Section: Review Of Related Workmentioning

confidence: 99%

“…Many methods have been proposed to solve the problem of automatic recognition of surgical procedures. In [11][12][13][14], the authors use instruments and sensor data directly to recognition of surgical activities. However, these methods require some special sensors and usually connected to a surgical instrument or a surgeon's hand, which may interfere with the normal operation of the operation.…”

Section: Introductionmentioning

confidence: 99%

Semi-supervised spatio-temporal CNN for recognition of surgical workflow

Chen

Sun

Zhong

2018

J Image Video Proc.

View full text Add to dashboard Cite

Robust and automated surgical workflow detection in real time is a core component of the future intelligent operating room. Based on this technology, it can help medical staff to automate and intelligently complete many routine activities during surgery. Recognition of surgical workflow based on traditional pattern recognition methods requires a large number of labeled surgical video data. However, the labeled surgical video data requires expert knowledge and it is difficult and time consuming to collect a sufficient number of labeled surgical video data in the medical field. Therefore, this paper proposes a semi-supervised spatio-temporal convolutional network for the recognition of surgical workflow based on convolutional neural networks and temporal-recursive networks. Firstly, we build a spatial convolutional extraction feature network based on unsupervised generative adversarial learning. Then, we build a bridge between lowlevel surgical video features and high-level surgical workflow semantics based on an unsupervised temporal-ordered network learning approach. Finally, we use the semi-supervised learning method to integrate the spatial model and the temporal model to fine-tune the network, and realize the intelligent recognition of the surgical workflow at a low cost to efficiently determine the progress of the surgical workflow. We performed some experiments for validating the mode based on m2cai16-workflow dataset. It shows that the proposed model can effectively extract the surgical feature and determine the surgical workflow. The Jaccard score of the model reaches 71.3%, and the accuracy of the model reaches 85.8%.

show abstract

Automatic data-driven real-time segmentation and recognition of surgical workflow

Abstract: Compared to the analysis based on one data type only, a combination of visual features and instrument signals allows better segmentation, reduction of the detection delay and discovery of the correct phase order.

Cited by 117 publications

References 21 publications

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Offline identification of surgical deviations in laparoscopic rectopexy

Semi-supervised spatio-temporal CNN for recognition of surgical workflow

Contact Info

Product

Resources

About