U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument

Abstract

Conventional therapy approaches limit surgeons' dexterity control due to limited field-of-view. With the advent of robot-assisted surgery, there has been a paradigm shift in medical technology for minimally invasive surgery. However, it is very challenging to track the position of the surgical instruments in a surgical scene, and accurate detection & identification of surgical tools is paramount. Deep learning-based semantic segmentation in frames of surgery videos has the potential to facilitate this task. In this work, we modify the U-Net architecture named U-NetPlus, by introducing a pre-trained encoder and re-design the decoder part, by replacing the transposed convolution operation with an upsampling operation based on nearest-neighbor (NN) interpolation. To further improve performance, we also employ a very fast and flexible data augmentation technique. We trained the framework on 8 × 225 frame sequences of robotic surgical videos, available through the MICCAI 2017 EndoVis Challenge dataset and tested it on 8 × 75 frame and 2 × 300 frame videos. Using our U-NetPlus architecture, we report a 90.20% DICE for binary segmentation, 76.26% DICE for instrument part segmentation, and 46.07% for instrument type (i.e., all instruments) segmentation, outperforming the results of previous techniques implemented and tested on these data.

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

“…We first compare our method with the state-of-the-art results in challenge on three tasks. Table 1 lists the performance of U-Net [14] (results quoted from [16]), TernausNet [16], and latest reported U-NetPlus [6] (an enhanced U-Net with batch normalized encoders and nearest neighbor interpolation). We see that MF-TAPNet consistently outperforms all other methods across all three tasks.…”

“…Our IoU exceeds the challenge winner by 3.96% at binary segmentation, 2.42% at part segmentation, and 2.84% at type segmentation. Though [16] and [6] develop advanced strategies to enhance a network, our method is superior by using temporal prior to explicitly provide a reliable guidance, which helps the network learn to focus on regions of interest. The improvement is more obvious for binary segmentation, because our prior also aggregates probabilities from all positive classes.…”

Brain Stroke Lesion Segmentation Using Consistent Perception Generative Adversarial Network

“…Fu et al [30] proposed a dual attention network to integrate local features with global dependencies, their appended the position attention module and channel attention module on the top of FCN and achieved good performance on three challenging scene segmentation dataset. Unfortunately, those skip connections demand the fusion of the same-scale encoder and decoder feature maps [31], and those methods is insensitive to the different sizes and locations of lesions. Inspired by these previous studies, we proposed a segmentation module using a similarity connection module to enhance the ability of the representation in our CPGAN.…”

Using conditional generative adversarial networks to reduce the effects of latency in robotic telesurgery

“…If the network is fed an image from the generator, and it decides that the image is a fake, the generator takes that feedback and adjusts its weights in order to produce a more realistic image. The generator is a U-Net architecture that builds a dense embedding of an image using convolutional layers and expands that embedding into a new generated image [14]. This process is shown in figure 2b.…”