Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Du, Xiaofei; Kurmann, Thomas; Chang, Ping-Lin; Allan, Maximilian; Ourselin, Sébastien; Sznitman, Raphael; Kelly, John D.; Stoyanov, Danail

doi:10.1109/tmi.2017.2787672

Cited by 116 publications

(115 citation statements)

References 30 publications

Supporting

Mentioning

112

Contrasting

Order By: Relevance

“…The method used a weighted cross-entropy loss to cope with the class imbalancing problem due to the small size of the target. [22] and [11] tracked surgical instruments using a deep network having an encoder-decoder architecture. Their approaches combined instrument segmentation and detection in a multi-task learning problem to make the tool detection in a cluttered background more robust.…”

Section: Interventional / Surgical Tool Trackingmentioning

confidence: 99%

Dynamic coronary roadmapping via catheter tip tracking in X-ray fluoroscopy with deep learning based Bayesian filtering

Smal

Daemen

et al. 2020

Medical Image Analysis

View full text Add to dashboard Cite

Percutaneous coronary intervention (PCI) is typically performed with image guidance using X-ray angiograms in which coronary arteries are opacified with X-ray opaque contrast agents. Interventional cardiologists typically navigate instruments using non-contrast-enhanced fluoroscopic images, since higher use of contrast agents increases the risk of kidney failure. When using fluoroscopic images, the interventional cardiologist needs to rely on a mental anatomical reconstruction. This paper reports on the development of a novel dynamic coronary roadmapping approach for improving visual feedback and reducing contrast use during PCI. The approach compensates cardiac and respiratory induced vessel motion by ECG alignment and catheter tip tracking in X-ray fluoroscopy, respectively. In particular, for accurate and robust tracking of the catheter tip, we proposed a new deep learning based Bayesian filtering method that integrates the detection outcome of a convolutional neural network and the motion estimation between frames using a particle filtering framework. The proposed roadmapping and tracking approaches were validated on clinical X-ray images, achieving accurate performance on both catheter tip tracking and dynamic coronary roadmapping experiments. In addition, our approach runs in real-time on a computer with a single GPU and has the potential to be integrated into the clinical workflow of PCI procedures, providing cardiologists with visual guidance during interventions without the need of extra use of contrast agent.

show abstract

Section: Interventional / Surgical Tool Trackingmentioning

confidence: 99%

Dynamic coronary roadmapping via catheter tip tracking in X-ray fluoroscopy with deep learning based Bayesian filtering

Smal

Daemen

et al. 2020

Medical Image Analysis

View full text Add to dashboard Cite

show abstract

“…There exist a number of algorithms to perform visionbased surgical tool tracking [3]. Also ConvNets have been shown to work well for this task [9,19]. In one study, Jin et al [16] use region-based ConvNets for instrument detection and localization.…”

mentioning

confidence: 99%

Video-based surgical skill assessment using 3D convolutional neural networks

et al. 2019

View full text Add to dashboard Cite

Purpose: A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill.Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. Methods: Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a Temporal Segment Network during training. Results: We evaluate the method on the publicly available JIGSAWS dataset, which consists of recordings of basic robot-assisted surgery tasks performed on a dry lab bench-top model. Our approach achieves high skill classification accuracies ranging from 95.1% to 100.0%. Conclusions: Our results demonstrate the feasibility of deep learning-based assessment of technical skill from surgical video. Notably, the 3D ConvNet is able to learn meaningful patterns directly from the data, alleviating the need for man-

show abstract

“…For the ablation study, inspired by [32], we compared the performance of the proposed framework with the detectiononly and regression-only architectures. Both were implemented in a spatio-temporal fashion (i.e., with 3D convolution).…”

Section: Ablation Study and Comparison With The State Of The Artmentioning

confidence: 99%

Preterm Infants’ Pose Estimation With Spatio-Temporal Features

Moccia

Migliorelli

Carnielli

et al. 2020

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

Objective: Preterm infants' limb monitoring in neonatal intensive care units (NICUs) is of primary importance for assessing infants' health status and motor/cognitive development. Herein, we propose a new approach to preterm infants' limb pose estimation that features spatio-temporal information to detect and track limb joints from depth videos with high reliability. Methods: Limb-pose estimation is performed using a deeplearning framework consisting of a detection and a regression convolutional neural network (CNN) for rough and precise joint localization, respectively. The CNNs are implemented to encode connectivity in the temporal direction through 3D convolution. Assessment of the proposed framework is performed through a comprehensive study with sixteen depth videos acquired in the actual clinical practice from sixteen preterm infants (the babyPose dataset). Results: When applied to pose estimation, the median root mean square distance, computed among all limbs, between the estimated and the ground-truth pose was 9.06 pixels, overcoming approaches based on spatial features only (11.27 pixels). Conclusion: Results showed that the spatio-temporal features had a significant influence on the pose-estimation performance, especially in challenging cases (e.g., homogeneous image intensity). Significance: This paper significantly enhances the state of art in automatic assessment of preterm infants' health status by introducing the use of spatio-temporal features for limb detection and tracking, and by being the first study to use depth videos acquired in the actual clinical practice for limb-pose estimation. The babyPose dataset has been released as the first annotated dataset for infants' pose estimation.

show abstract

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Cited by 116 publications

References 30 publications

Dynamic coronary roadmapping via catheter tip tracking in X-ray fluoroscopy with deep learning based Bayesian filtering

Dynamic coronary roadmapping via catheter tip tracking in X-ray fluoroscopy with deep learning based Bayesian filtering

Video-based surgical skill assessment using 3D convolutional neural networks

Preterm Infants’ Pose Estimation With Spatio-Temporal Features

Contact Info

Product

Resources

About