Addressing multi-label imbalance problem of surgical tool detection using CNN

Sahu, Manish; Mukhopadhyay, Anirban; Szengel, Angelika; Zachow, Stefan

doi:10.1007/s11548-017-1565-x

Cited by 41 publications

(41 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wang et al (2017) integrated VGG and GoogleNet to take advantage of the deep CNN model ensemble. Sahu et al (2017) paid attention to analyzing the imbalance on tool co-occurrences and exploited stratification techniques during the network training process. Choi et al (2017) developed a real-time detection CNN model based on YOLO.…”

Section: Surgical Video Analysismentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

158

View full text Add to dashboard Cite

Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis and also very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both tasks. Specifically, our proposed MTRCNet-CL model has an end-to-end architecture with two branches, which share earlier feature encoders to extract general visual features while holding respective higher layers targeting for specific tasks. Given that temporal information is crucial for phase recognition, long-short term memory (LSTM) is explored to model the sequential dependencies in the phase recognition branch. More importantly, a novel and effective correlation loss is designed to model the relatedness between tool presence and phase identification of each video frame, by minimizing the divergence of predictions from the two branches. Mutually leveraging both lowlevel feature sharing and high-level prediction correlating, our MTRCNet-CL method can encourage the interactions between the two tasks to a large extent, and hence can bring about benefits to each other. Extensive experiments on a large surgical video dataset (Cholec80) demonstrate outstanding performance of our proposed method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 89.1% v.s. 81.0% for the mAP in tool presence detection and 87.4% v.s. 84.5% for F1 score in phase recognition). The code can be found on our project website.

show abstract

Section: Surgical Video Analysismentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

158

View full text Add to dashboard Cite

show abstract

“…In the past few years, it has been applied to medical image datasets and deep networks have been developed for various medical applications such as segmentation [18] or recognition tasks [19] . The methodology has been demonstrated to be effective in instrument presence detection [20] or localization [21] . Additionally, networks for semantic instrument segmentation have also been proposed and shown to be effective in real-time performance [22] , [23] .…”

Section: Introductionmentioning

confidence: 99%

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Kurmann

Chang³

et al. 2018

IEEE Trans. Med. Imaging

113

111

View full text Add to dashboard Cite

Instrument detection, pose estimation, and tracking in surgical videos are an important vision component for computer-assisted interventions. While significant advances have been made in recent years, articulation detection is still a major challenge. In this paper, we propose a deep neural network for articulated multi-instrument 2-D pose estimation, which is trained on detailed annotations of endoscopic and microscopic data sets. Our model is formed by a fully convolutional detection-regression network. Joints and associations between joint pairs in our instrument model are located by the detection subnetwork and are subsequently refined through a regression subnetwork. Based on the output from the model, the poses of the instruments are inferred using maximum bipartite graph matching. Our estimation framework is powered by deep learning techniques without any direct kinematic information from a robot. Our framework is tested on single-instrument RMIT data, and also on multi-instrument EndoVis and in vivo data with promising results. In addition, the data set annotations are publicly released along with our code and model.

show abstract

“…Another way to address class imbalance would be to use a loss function that calculates the weighted sum of the losses of each individual class based on the class frequency [ 23 ]. Alternatively, we could apply stratification techniques to make sure all classes are parsed when training the model [ 24 ]. Although these methods were not explored in the Letter we will be exploring their application to address class imbalance in future works.…”

Section: Resultsmentioning

confidence: 99%

Can surgical simulation be used to train detection and classification of neural networks?

Zisimopoulos

Flouty

Stacey

et al. 2017

Healthc. technol. lett.

View full text Add to dashboard Cite

Computer-assisted interventions (CAI) aim to increase the effectiveness, precision and repeatability of procedures to improve surgical outcomes. The presence and motion of surgical tools is a key information input for CAI surgical phase recognition algorithms. Vision-based tool detection and recognition approaches are an attractive solution and can be designed to take advantage of the powerful deep learning paradigm that is rapidly advancing image recognition and classification. The challenge for such algorithms is the availability and quality of labelled data used for training. In this Letter, surgical simulation is used to train tool detection and segmentation based on deep convolutional neural networks and generative adversarial networks. The authors experiment with two network architectures for image segmentation in tool classes commonly encountered during cataract surgery. A commercially-available simulator is used to create a simulated cataract dataset for training models prior to performing transfer learning on real surgical data. To the best of authors’ knowledge, this is the first attempt to train deep learning models for surgical instrument detection on simulated data while demonstrating promising results to generalise on real data. Results indicate that simulated data does have some potential for training advanced classification methods for CAI systems.

show abstract

Addressing multi-label imbalance problem of surgical tool detection using CNN

Abstract: The analysis on tool imbalance, backed by the empirical results, indicates the need and superiority of the proposed framework over state-of-the-art techniques.

Cited by 41 publications

References 13 publications

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Can surgical simulation be used to train detection and classification of neural networks?

Contact Info

Product

Resources

About