An Anchor-Free Convolutional Neural Network for Real-Time Surgical Tool Detection in Robot-Assisted Surgery

Abstract

Robot-assisted surgery (RAS), a type of minimally invasive surgery, is used in a variety of clinical surgeries because it has a faster recovery rate and causes less pain. Automatic video analysis of RAS is an active research area, where precise surgical tool detection in real time is an important step. However, most deep learning methods currently employed for surgical tool detection are based on anchor boxes, which results in low detection speeds. In this paper, we propose an anchor-free convolutional neural network (CNN) architecture, a novel frame-by-frame method using a compact stacked hourglass network, which models the surgical tool as a single point: the center point of its bounding box. Our detector eliminates the need to design a set of anchor boxes, and is end-to-end differentiable, simpler, more accurate, and more efficient than anchor-box-based detectors. We believe our method is the first to incorporate the anchor-free idea for surgical tool detection in RAS videos. Experimental results show that our method achieves 98.5% mAP and 100% mAP at 37.0 fps on the ATLAS Dione and Endovis Challenge datasets, respectively, and truly realizes real-time surgical tool detection in RAS videos.

Real-Time Surgical Tool Detection in Minimally Invasive Surgery Based on Attention-Guided Convolutional Neural Network

“…Our method achieved a different detection mAP on the two public datasets because the ATLAS Dione dataset is more challenging due to various disturbing factors, such as different surgical tasks, motion blurring, high deformation, tools with occlusion, tools' overlap, and missing tool parts. With the detection mAP of 100 and 94.05% on the EndoVis Challenge and ATLAS Dione datasets, respectively, our method surpassed all methods except CenterNet [24]. Our method had the highest speed (55.5 FPS) on the two public datasets, which could satisfy the realtime requirement of STD in MIS.…”

“…As shown in Table 3, we also provided the average precision (AP) per tool of all methods on the Cholec80-locations dataset to further prove our method's feasibility. As Table 3 shows, our method achieved the mAP of 91.65% on the new dataset, which was slightly lower than CenterNet [24], but outperformed all other detection methods. The AP values of all surgical tools, except grasper and irrigator, exceeded 90%.…”

“…Under the two evaluation metrics, we separately compared our proposed method's results to those of eight state-of-the-art detection methods on the three MIS datasets introduced above. The eight state-of-the-art detection methods included three two-stage methods (Faster RCNN [21], RelationNet [35] and Cascade CNN [23]) and five single-stage methods (Retinanet [36], Yolov3 [37], HR-Net [38], CenterNet [24] and RefineDet [28]). The detection results for all methods on the two public datasets and the Cholec80-locations dataset are summarized in Tables 2 and 3, respectively.…”

“…Our method inherited the above advantages and further improved the detection performance by using the attention mechanism and light-head modules specially designed to detect the surgical tool in MIS. In terms of accuracy, our method sur-passed all methods except CenterNet [24], but had the highest speed (55.5 FPS) that could satisfy the real-time requirement of STD in MIS. Compared to Basic Network in Table 4, our method's accuracy was improved by 2.43 and 2.24%, respectively, on the ATLAS Dione and Cholec80-locations datasets, which was largely attributed to the introduction of the attention mechanism.…”

“…This algorithm achieved better detection performance in terms of detection accuracy and speed, but it failed to achieve end-toend training. Later, Liu et al [24] proposed an anchor-free CNN architecture by using a lightweight stacked hourglass network, which modelled the surgical tool as a single point: the center point of its bounding-box. This method eliminated the need to design a set of anchor boxes, and achieved endto-end training.…”