Temporal Coherence-based Self-supervised Learning for Laparoscopic Workflow Analysis

Abstract

In order to provide the right type of assistance at the right time, computer-assisted surgery systems need context awareness. To achieve this, methods for surgical workflow analysis are crucial. Currently, convolutional neural networks provide the best performance for video-based workflow analysis tasks. For training such networks, large amounts of annotated data are necessary. However, collecting a sufficient amount of data is often costly, time-consuming, and not always feasible. In this paper, we address this problem by presenting and comparing different approaches for self-supervised pretraining of neural networks on unlabeled laparoscopic videos using temporal coherence. We evaluate our pretrained networks on Cholec80, a publicly available dataset for surgical phase segmentation, on which a maximum F1 score of 84.6 was reached. Furthermore, we were able to achieve an increase of the F1 score of up to 10 points when compared to a non-pretrained neural network.

Real-Time Surgical Tool Detection in Minimally Invasive Surgery Based on Attention-Guided Convolutional Neural Network

“…For example, one can use the recurrent neural network (RNN) or its variants to extract the MIS videos' long-term temporal information and model the temporal dependency between the frames to get better results. Moreover, there are limited datasets with location annotations of tools, the weakly supervised [39], [40] or self-supervised methods [41]- [43] can be used to reduce the dependency on spatially annotated data. To the best of our knowledge, the Cholec80 dataset [31] contains various real-world environments, so we will continue to label the Cholec80 dataset with the coordinates of bounding boxes of surgical tools to address the lack of public datasets and further promote the development of STD in MIS.…”

Active learning using deep Bayesian networks for surgical workflow analysis

“…As previously stated, we evaluate our proposed active learning methods for surgical workflow analysis tasks (instrument presence and phase segmentation) on the Cholec80 dataset. For this, we first divide the dataset in 4 subsets of 20 videos each, as outlined in [6]. Each video was sampled at a rate of one frame per second and each frame was downsampled to a resolution of 384×216 pixels.…”

“…Long short-term memory units (LSTM), a more complex form of the RNN, can learn to strategically remember, but also forget, information from previ- ously seen inputs, while forgoing the problem of vanishing gradients common to RNNs [11]. Combining CNNs with LSTMs makes video-based workflow analysis, by using exclusively deep neural networks, possible [2,3,6,23].…”

Artificial Intelligence-Assisted Surgery: Potential and Challenges

“…Such methods can be used for workflow analysis through automatic segmentation of procedures into phases or surgical actions. These methods rely, for example, on data from tool usage [28, 29] or robotic kinematics [30] or, by using DL, directly on camera data, such from an endoscope [20, 31, 32] or 3D camera [32]. DL methods for determining surgery duration (Fig.…”