Real-time tracking of surgical instruments based on spatio-temporal context and deep learning

Abstract

Real-time tool tracking in minimally invasive-surgery (MIS) has numerous applications for computer-assisted interventions (CAIs). Visual tracking approaches are a promising solution to realtime surgical tool tracking, however, many approaches may fail to complete tracking when the tracker suffers from issues such as motion blur, adverse lighting, specular reflections, shadows, and occlusions. We propose an automatic real-time method for two-dimensional tool detection and tracking based on a spatial transformer network (STN) and spatio-temporal context (STC). Our method exploits both the ability of a convolutional neural network (CNN) with an in-house trained STN and STC to accurately locate the tool at high speed. Then we compared our method experimentally with other four general of CAIs' visual tracking methods using eight existing online and in-house datasets, covering both in vivo abdominal, cardiac and retinal clinical cases in which different surgical instruments were employed. The experiments demonstrate that our method achieved great performance with respect to the accuracy and the speed. It can track a surgical tool without labels in real time in the most challenging of cases, with an accuracy that is equal to and sometimes surpasses most state-of-the-art tracking algorithms. Further improvements to our method will focus on conditions of occlusion and multi-instruments.

An Anchor-Free Convolutional Neural Network for Real-Time Surgical Tool Detection in Robot-Assisted Surgery

“…Image-based methods are becoming more popular, as they rely purely on equipment already in the operating theatre [8]. Deep convolutional neural network (CNN) has been merged into various RAS medical image-based tasks, such as surgical tool detection [9]- [13], tracking [14]- [18], pose estimation [19]- [21], and segmentation [22]- [25]. Singleand two-stage detectors are generally used to detect surgical tools.…”

Hey Siri! Perform a type 3 hysterectomy. Please watch out for the ureter!” What is autonomous surgery and what are the latest developments?

“…Real-time instrument tracking is getting faster and more precise, thanks to highly powerful computer processors and deep learning algorithms that successfully determine spatial location. The success rate can be increased further by linking the instruments to be used to machine learning systems and pre-educating them before the operation ( 6 ).…”

Intraoperative surgery room management: A deep learning perspective

“…Understand not only the presence but also the position gives great improvement in phase recognition. This can evolve in tracking, 35,36 and segmentation, 37‐41 a technique to define the contours of all the different tools in the current frame, shown in Figure 4C. All these discussed studies: Used supervised learning; conversely, in References 42 and 43, the authors applied unsupervised learning to assign phases to specific frame and avoid the time‐consuming labelling phase. Focused on endoscopic videos; same techniques have been applied to external cataracts video, in References 44 and 45.…”