Real-Time Segmentation of Non-rigid Surgical Tools Based on Deep Learning and Tracking

Abstract

Abstract. Real-time tool segmentation is an essential component in computer-assisted surgical systems. We propose a novel real-time automatic method based on Fully Convolutional Networks (FCN) and optical flow tracking. Our method exploits the ability of deep neural networks to produce accurate segmentations of highly deformable parts along with the high speed of optical flow. Furthermore, the pre-trained FCN can be fine-tuned on a small amount of medical images without the need to hand-craft features. We validated our method using existing and new benchmark datasets, covering both ex vivo and in vivo real clinical cases where different surgical instruments are employed. Two versions of the method are presented, non-real-time and real-time. The former, using only deep learning, achieves a balanced accuracy of 89.6% on a real clinical dataset, outperforming the (non-real-time) state of the art by 3.8 percentage points. The latter, a combination of deep learning with optical flow tracking, yields an average balanced accuracy of 78.2% across all the validated datasets.

CFCM: Segmentation via Coarse to Fine Context Memory

“…As specified in the guidelines, we preformed a cross-validation by leaving one surgery out of the training data. The segmentation result was compared to generic methods as well as algorithms that were explicitly published for this task [14,7,18]. Garcia P. Herrera et al [7] proposed a Fully convolutional network for segmentation in minimally invasive surgery.…”

“…The segmentation result was compared to generic methods as well as algorithms that were explicitly published for this task [14,7,18]. Garcia P. Herrera et al [7] proposed a Fully convolutional network for segmentation in minimally invasive surgery. To achieve real-time performance, they applied the network only on every couple of frames and propagated the information with optical flow (FCN+OF).…”

GIFT-Grab: Real-time C++ and Python multi-channel video capture, processing and encoding API

“…Name: Zenodo [10,9]. Other examples include real-time stereo reconstruction [48] and probe tracking [49] in robotic surgery, surveillance endoscopies [50], real-time panorama image synthesis [51], vehicle surveillance [52] and content-based video identification [53].…”

“…GIFT-Surg involves major innovations in science and engineering combined with clinical translation for improving fetal therapy and diagnosis by providing advanced image processing and visualisation capabilities. We have already obtained promising results with a number of novel methods including real-time mosaicing of the placenta [9] and sensor-free real-time instrument tracking [10]. These methods rely on real-time video streams from medical devices such as endoscopes and ultrasound probes.…”

RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments

“…Recently, a serious of methods have been proposed to segment surgical instruments. Luis et al [3] presented a network based on Fully Convolutional Networks(FCN) and optic flow to solve problems such as occlusion and deformation of surgical instruments. RASNet [4] adopted an attention module to emphasize the targets region and improve the feature representation.…”