2D/3D Real-Time Tracking of Surgical Instruments Based on Endoscopic Image Processing

Abstract

International audienceThis paper describes a simple and robust algorithm which permits to track surgical instruments without artificial markers in endoscopic images. Based on image processing, this algorithm can estimate the 2D/3D pose of all the instruments visible in the image, in real-time (30 Hz). The originality of the approach is based on the use of a Frangi filter for detecting edges and the tip of instruments. The accuracy of the instruments’ location in the image is evaluated using an extensive dataset (1500 images, 3 laparoscopic surgeries). Pose estimation of instruments in space is quantitatively evaluated on a test bench through comparison with the ground truth positioning provided by a calibrated robotic instrument holder. This method opens perspectives in the real-time control of surgical robots and the intra-operative recognition of surgical gestures

Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method

“…The comparison of 3D trajectories in Figure 5 and Table 1 proves their close fit: the mean errors in X-Y-Z directions are less than 3 mm, while their standard deviations imply the proposed method's strong robustness to noise interference. As shown in Figure 6, the maximal 3D error of the proposed method is about 5 mm, which is even less than the mean error in [9]. The collinear hypothesis used in the Equation (4) accounts for the small errors in X-Y coordinates: insofar as the insertion point is included in the estimation of tracking position, the errors of Z coordinates are well-controlled (less than 1 mm).…”

“…These methods are accurate in tracking and efficient in computation, but suffer from unresolved issues of color and lighting variations. Some other vision-based methods exploit the geometric constraints [8] and the gradientlike features [9,10], in order to identify the shaft of instrument, but fail to provide more accurate 3D positions of the instrument tip. Machine learning techniques [11][12][13][14][15][16][17][18][19] introduced into the instrument detection and tracking provide training of their discriminative classifiers/models according to the input visual features of the foreground (instrument tip or shaft).…”

Convolutional neural network-based surgical instrument detection

“…Conventional methods use low-level features to locate instruments in images [4]. For instance, Augustine and Voros [5] proposed a simple and robust algorithm to estimate the 2D/3D pose of instruments based on image processing. Another example is the study by Rieke et al [6], which employed the RGB and HOG feature of the instrument for the detection task.…”

Real-time tracking of surgical instruments based on spatio-temporal context and deep learning

“…Certain instruments that employ robot kinematic information from joint encoders to track the movements of an entire surgical instrument [4] are typically fast and robust, since they can recover when the instruments are occluded by tissue, smoke or shadows from other instruments, however, the accumulation of errors can result in a significantly large aggregate error [5]. The method of extracting image features with gradients [6] can accurately render partial surgical tool features, however, it has two disadvantages: one being that gradients are sensitive to noise, illumination and shadows [7], while the other is that the gradient distribution rate is difficult to obtain [8]. In addition, the methods of template matching by learning [9,10], can accurately extract features from image patches and quickly and robustly track the instruments, since they use a paradigm of pre-training to carry out image matching.…”