Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Jin, Yueming; Cheng, Kuo‐Sheng; Dou, Qi; Heng, Pheng‐Ann

doi:10.1007/978-3-030-32254-0_49

Cited by 97 publications

(69 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Research on instrument segmentation for endoscopic procedures is dominated by supervision-based approaches ranging from full supervision [5], semi/self-supervision [25], and weak supervision [12] up to multi-task [16] and multi-modal learning [15]. Some recent works also explored unsupervised approaches [7,18], however, for the sake of brevity, we will only focus on approaches that employ learning from simulation data for unsupervised domain adaptation.…”

Section: Related Workmentioning

confidence: 99%

Simulation-to-real domain adaptation with teacher–student learning for endoscopic instrument segmentation

2021

View full text Add to dashboard Cite

Purpose Segmentation of surgical instruments in endoscopic video streams is essential for automated surgical scene understanding and process modeling. However, relying on fully supervised deep learning for this task is challenging because manual annotation occupies valuable time of the clinical experts. Methods We introduce a teacher–student learning approach that learns jointly from annotated simulation data and unlabeled real data to tackle the challenges in simulation-to-real unsupervised domain adaptation for endoscopic image segmentation. Results Empirical results on three datasets highlight the effectiveness of the proposed framework over current approaches for the endoscopic instrument segmentation task. Additionally, we provide analysis of major factors affecting the performance on all datasets to highlight the strengths and failure modes of our approach. Conclusions We show that our proposed approach can successfully exploit the unlabeled real endoscopic video frames and improve generalization performance over pure simulation-based training and the previous state-of-the-art. This takes us one step closer to effective segmentation of surgical instrument in the annotation scarce setting.

show abstract

Section: Related Workmentioning

confidence: 99%

Simulation-to-real domain adaptation with teacher–student learning for endoscopic instrument segmentation

2021

View full text Add to dashboard Cite

show abstract

“…Such MOT techniques generally rely on a real-time highly accurate detector being run at every frame, while a tracker manager ensures long-term trajectories with optimal data assignment. Promising results have been recently reported in surgical instrument segmentation [12,8,17,7] which could greatly contribute towards an improved multiple object tracker.…”

Section: Discussionmentioning

confidence: 99%

Towards real-time multiple surgical tool tracking

Robu

Kadkhodamohammadi

Luengo

et al. 2020

Computer Methods in Biomechanics and Biomedical Engineering: Im

View full text Add to dashboard Cite

Surgical tool tracking is an essential building block for computer assisted interventions (CAI) and applications like video summarization, workflow analysis and surgical navigation. Vision-based instrument tracking in laparoscopic surgical data faces significant challenges such as fast instrument motion, multiple simultaneous instruments, and re-initialization due to out-of-view conditions or instrument occlusions. In this paper, we propose a real-time multiple object tracking framework for whole laparoscopic tools, which extends an existing single object tracker. We introduce a geometric object descriptor, which helps with overlapping bounding box disambiguation, fast motion and optimal assignment between existing trajectories and new hypotheses. We achieve 99.51% and 75.64% average accuracy on ex-vivo robotic data and in-vivo laparoscopic sequences respectively from the Endovis'15 Instrument Tracking Dataset. The proposed geometric descriptor increased the performance on laparoscopic data by 32%, significantly reducing identity switches, false negatives and false positives. Overall, the proposed pipeline can successfully recover trajectories over long-sequences and it runs in real-time at approximately 25-29 fps.

show abstract

“…A central topic in these applications is the correct identification of surgical instruments, where the main focus so far has been the segmentation of the instruments [2,7,9,13]. These methods have shown promising performance for binary segmentation, but have under-performed in instrument type segmentation tasks.…”

Section: Introductionmentioning

confidence: 99%

Mask then classify: multi-instance segmentation for surgical instruments

et al. 2021

View full text Add to dashboard Cite

Purpose The detection and segmentation of surgical instruments has been a vital step for many applications in minimally invasive surgical robotics. Previously, the problem was tackled from a semantic segmentation perspective, yet these methods fail to provide good segmentation maps of instrument types and do not contain any information on the instance affiliation of each pixel. We propose to overcome this limitation by using a novel instance segmentation method which first masks instruments and then classifies them into their respective type. Methods We introduce a novel method for instance segmentation where a pixel-wise mask of each instance is found prior to classification. An encoder–decoder network is used to extract instrument instances, which are then separately classified using the features of the previous stages. Furthermore, we present a method to incorporate instrument priors from surgical robots. Results Experiments are performed on the robotic instrument segmentation dataset of the 2017 endoscopic vision challenge. We perform a fourfold cross-validation and show an improvement of over 18% to the previous state-of-the-art. Furthermore, we perform an ablation study which highlights the importance of certain design choices and observe an increase of 10% over semantic segmentation methods. Conclusions We have presented a novel instance segmentation method for surgical instruments which outperforms previous semantic segmentation-based methods. Our method further provides a more informative output of instance level information, while retaining a precise segmentation mask. Finally, we have shown that robotic instrument priors can be used to further increase the performance.

show abstract

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Cited by 97 publications

References 21 publications

Simulation-to-real domain adaptation with teacher–student learning for endoscopic instrument segmentation

Simulation-to-real domain adaptation with teacher–student learning for endoscopic instrument segmentation

Towards real-time multiple surgical tool tracking

Mask then classify: multi-instance segmentation for surgical instruments

Contact Info

Product

Resources

About