Real‐time surgical instrument detection in robot‐assisted surgery using a convolutional neural network cascade

Abstract

Surgical instrument detection in robot-assisted surgery videos is an import vision component for these systems. Most of the current deep learning methods focus on single-tool detection and suffer from low detection speed. To address this, the authors propose a novel frameby-frame detection method using a cascading convolutional neural network (CNN) which consists of two different CNNs for real-time multi-tool detection. An hourglass network and a modified visual geometry group (VGG) network are applied to jointly predict the localisation. The former CNN outputs detection heatmaps representing the location of tool tip areas, and the latter performs bounding-box regression for tool tip areas on these heatmaps stacked with input RGB image frames. The authors' method is tested on the publicly available EndoVis Challenge dataset and the ATLAS Dione dataset. The experimental results show that their method achieves better performance than mainstream detection methods in terms of detection accuracy and speed.

An Anchor-Free Convolutional Neural Network for Real-Time Surgical Tool Detection in Robot-Assisted Surgery

“…Image-based methods are becoming more popular, as they rely purely on equipment already in the operating theatre [8]. Deep convolutional neural network (CNN) has been merged into various RAS medical image-based tasks, such as surgical tool detection [9]- [13], tracking [14]- [18], pose estimation [19]- [21], and segmentation [22]- [25]. Singleand two-stage detectors are generally used to detect surgical tools.…”

“…Singleand two-stage detectors are generally used to detect surgical tools. Two-stage detectors [1], [3] apply a region proposal network (RPN) to generate region proposals before being passed to a final classification and bounding box refinement network; single-stage detectors [13], [26] place anchor boxes densely over an image and generate final box predictions by scoring anchor boxes and refining their coordinates through regression. Both single-and two-stage detectors use anchor boxes extensively, but single-stage detectors are more competitive and efficient than two-stage detectors.…”

“…The method proposed by Jin et al [5] and the surgical tool detection method proposed by Twinanda et al [9] only detect the presence of the tool and cannot output the location of the surgical tool. Zhao et al [13] presented a CNN-cascaded surgical tool detection method, which can not achieve end-to-end training and needs to design the output heatmaps carefully. Compared with the work of Zhao et al [13], our method can not only achieve end-to-end training, but also innovatively use a more efficient and compact CNN backbone, which has an accuracy rate that exceeds their work at comparable speeds.…”

“…Zhao et al [13] presented a CNN-cascaded surgical tool detection method, which can not achieve end-to-end training and needs to design the output heatmaps carefully. Compared with the work of Zhao et al [13], our method can not only achieve end-to-end training, but also innovatively use a more efficient and compact CNN backbone, which has an accuracy rate that exceeds their work at comparable speeds.…”

Real-Time Surgical Tool Detection in Minimally Invasive Surgery Based on Attention-Guided Convolutional Neural Network

“…This method achieved good detection accuracy of surgical tools, but Zhang et al [22] did not disclose the detection speed. Zhao et al [23] proposed using a cascade CNN to perform STD, which consisted of an hourglass network and a modified visual geometry group (VGG) network. This algorithm achieved better detection performance in terms of detection accuracy and speed, but it failed to achieve end-toend training.…”