Real-Time Online Adaption for Robust Instrument Tracking and Pose Estimation

Rieke, Nicola; Tan, David Joseph; Tombari, Federico; Vizcaíno, Josué Page; Filippo, Chiara Amat di San; Eslami, Abouzar; Navab, Nassir

doi:10.1007/978-3-319-46720-7_49

Cited by 16 publications

(17 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Image-based surgical instrument detection and tracking is attractive because it relies purely on equipment already in the operating theatre [4] . Likewise pose estimation from images has been shown to be feasible in different specialisations, such as retinal microsurgery [5] – [7] , neurosurgery [8] and MIS [9] – [11] . While both detection and tracking are difficult, pose estimation presents additional challenges due to the complex articulation structure.…”

Section: Introductionmentioning

confidence: 99%

“…While both detection and tracking are difficult, pose estimation presents additional challenges due to the complex articulation structure. Most image-based methods [7] , [11] often extract low-level visual features from keypoints or regions to learn offline or online part appearance templates by using machine learning algorithms. Such low-level feature representations usually suffer from a lack of semantic interpretation, which means they cannot capture the high level category appearance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Kurmann

Chang³

et al. 2018

IEEE Trans. Med. Imaging

113

111

View full text Add to dashboard Cite

Instrument detection, pose estimation, and tracking in surgical videos are an important vision component for computer-assisted interventions. While significant advances have been made in recent years, articulation detection is still a major challenge. In this paper, we propose a deep neural network for articulated multi-instrument 2-D pose estimation, which is trained on detailed annotations of endoscopic and microscopic data sets. Our model is formed by a fully convolutional detection-regression network. Joints and associations between joint pairs in our instrument model are located by the detection subnetwork and are subsequently refined through a regression subnetwork. Based on the output from the model, the poses of the instruments are inferred using maximum bipartite graph matching. Our estimation framework is powered by deep learning techniques without any direct kinematic information from a robot. Our framework is tested on single-instrument RMIT data, and also on multi-instrument EndoVis and in vivo data with promising results. In addition, the data set annotations are publicly released along with our code and model.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Kurmann

Chang³

et al. 2018

IEEE Trans. Med. Imaging

113

111

View full text Add to dashboard Cite

show abstract

“…The top-down feature fusion path is constructed by laterally connecting multi-layers of the backbone. The top layer 5 P of the top-down path is generated from the corresponding output 5 C with a 1×1 convolutional. A 2×2 deconvolutional layer with a stride of 2 is applied to 5 P , which outputs a feature map that has the same size as 4 C .…”

Section: Tools Detection and Bounding Box Regressionmentioning

confidence: 99%

“…Earlier surgical tool detection methods mostly extract low-level visual features [5], [6], such as color, gradient, and texture. However, these methods are inefficient and error prone that small or large tools may be overlooked.…”

Section: Introductionmentioning

confidence: 99%

FSNet: Pose Estimation of Endoscopic Surgical Tools Using Feature Stacked Network

2019

Proceedings of 2019 the 9th International Workshop on Computer Science and Engineering

View full text Add to dashboard Cite

Identification of surgical instruments is important to understand surgical scenarios and provide assistant processing in endoscopic image-guided surgery. In this paper, we propose a novel feature stacked network (FSNet) for the recognition of surgical tools in endoscopic images. With a lateral connection and concatenation operation on the different layers of the feature pyramid network, high-level semantic information is fused to low-level features, and the bounding boxes are regressed for the tool instance proposals. Then, low-level semantic information is propagated to a high-level network through the bottom-up feature concatenating path. The keypoints of tools are detected in each proposed boundary box. Two state-ofthe-art end-to-end tool keypoint recognition networks and three backbones are implemented for comparison. The AP and AR of the our FSNet based on ResNeXt101 are 46.1% and 36.5%, respectively, which surpass the results of other methods.

show abstract

“…4. RM dataset: Comparison to FPBC [23], POSE [3] and Online Adaption [9], measured by the metric KBB. The charts (a) to (c) show the accuracy for the left tip, right tip and center joint, respectively, for the Half Split experiment.…”

Section: Evaluation Of Modeling Strategiesmentioning

confidence: 99%

Concurrent Segmentation and Localization for Tracking of Surgical Instruments

Laina

Rieke

Rupprecht

et al. 2017

Lecture Notes in Computer Science

Self Cite

125

View full text Add to dashboard Cite

Real-time instrument tracking is a crucial requirement for various computer-assisted interventions. In order to overcome problems such as specular reflections and motion blur, we propose a novel method that takes advantage of the interdependency between localization and segmentation of the surgical tool. In particular, we reformulate the 2D instrument pose estimation as heatmap regression and thereby enable a concurrent, robust and near real-time regression of both tasks via deep learning. As demonstrated by our experimental results, this modeling leads to a significantly improved performance than directly regressing the tool position and allows our method to outperform the state of the art on a Retinal Microsurgery benchmark and the MICCAI EndoVis Challenge 2015. * I. Laina and N. Rieke contributed equally to this work.

show abstract

Real-Time Online Adaption for Robust Instrument Tracking and Pose Estimation

Cited by 16 publications

References 15 publications

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

FSNet: Pose Estimation of Endoscopic Surgical Tools Using Feature Stacked Network

Concurrent Segmentation and Localization for Tracking of Surgical Instruments

Contact Info

Product

Resources

About