TeCNO: Surgical Phase Recognition with Multi-stage Temporal Convolutional Networks

Czempiel, Tobias; Paschali, Magdalini; Keicher, Matthias; Simson, Walter; Feußner, H.; Kim, Seong Tae; Navab, Nassir

doi:10.1007/978-3-030-59716-0_33

Cited by 138 publications

(129 citation statements)

References 19 publications

Supporting

Mentioning

125

Contrasting

Order By: Relevance

“…MS-TCN [9] consisted of a multi-stage predictor architecture with each stage consisting of multi-layer TCN, which incrementally refined the prediction of the previous stage. Recently, TeCNO [7] adapted the MS-TCN architecture for online surgical phase prediction by implementing causal convolutions [23]. We build upon this architecture and confirm experimentally that it is superior to LSTM for multi-level activity recognition.…”

Section: Related Workmentioning

confidence: 64%

“…4. Feature Extraction Architecture ResNet-50 [13] has been successfully employed in many works for phase segmentation [7,15,16,28]. In this work, we utilize the same architecture as our backbone visual feature extraction model.…”

Section: Hierarchical Surgical Activities: Phases and Stepsmentioning

confidence: 99%

“…Our temporal model consists of only temporal convolutional layers; in particular, they are dilated residual layers performing dilated convolutions. Since our aim is to segment surgical activities online, similar to TeCNO [7], we perform causal convolutions [23] at each layer which depends only on n past frames and does not rely on any future frames. The dilation factor is increased by a factor of 2 for each consecutive layer which increases exponentially the temporal receptive field of the network without introducing any pooling layer.…”

Section: Temporal Modelingmentioning

confidence: 99%

“…Our models were trained on NVIDIA GeForce RTX 2080 Ti GPUs. Evaluation Metrics We follow the same evaluation metrics used in other related publications [7,15,16], where accuracy (ACC), precision (PR), recall (RE), and F1 score (F1) are used to effectively compare the results. Accuracy quantifies the total correct classification of activity in the whole video.…”

Section: Model Trainingmentioning

confidence: 99%

“…Recent works have strongly focused on developing methods to recognize one specific level of granularity from video data. The visual detection of phases [7,15,16,25,30], robotic gestures [2,10,26,29], and instruments [11,14,16,22] has, for instance, seen a surge in research activities, due to their potential impact on developing intra-and postoperative tools for the purposes of monitoring safety, assessing skills, and reporting. Many of these previous works have focused on endoscopic cholecystectomy procedures, utilizing the publicly available large-scale Cholec80 dataset [25], and on cataract surgical procedures, utilizing the popular CATARACTS dataset [11,30].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multi-task temporal convolutional networks for joint recognition of surgical phases and steps in gastric bypass procedures

et al. 2021

View full text Add to dashboard Cite

Purpose Automatic segmentation and classification of surgical activity is crucial for providing advanced support in computer-assisted interventions and autonomous functionalities in robot-assisted surgeries. Prior works have focused on recognizing either coarse activities, such as phases, or fine-grained activities, such as gestures. This work aims at jointly recognizing two complementary levels of granularity directly from videos, namely phases and steps. Methods We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a multi-task multi-stage temporal convolutional network (MTMS-TCN) along with a multi-task convolutional neural network (CNN) training setup to jointly predict the phases and steps and benefit from their complementarity to better evaluate the execution of the procedure. We evaluate the proposed method on a large video dataset consisting of 40 surgical procedures (Bypass40). Results We present experimental results from several baseline models for both phase and step recognition on the Bypass40. The proposed MTMS-TCN method outperforms single-task methods in both phase and step recognition by 1-2% in accuracy, precision and recall. Furthermore, for step recognition, MTMS-TCN achieves a superior performance of 3-6% compared to LSTM-based models on all metrics. Conclusion In this work, we present a multi-task multi-stage temporal convolutional network for surgical activity recognition, which shows improved results compared to single-task models on a gastric bypass dataset with multi-level annotations. The proposed method shows that the joint modeling of phases and steps is beneficial to improve the overall recognition of each type of activity.

show abstract

Section: Related Workmentioning

confidence: 64%