Surgical instrument segmentation for endoscopic vision with data fusion of cnn prediction and kinematic pose

Qin, Fangbo; Li, Yangming; Su, Yun-Hsuan; Xu, De; Hannaford, Blake

doi:10.1109/icra.2019.8794122

Cited by 48 publications

(25 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Early approaches used these features to assist in tool detection and segmentation using support vector machines [ 54 , 57 ] and decision forests [ 50 ]. More recently, neural network and UNet-based methods have emerged as promising directions [ 58 , 59 , 60 , 61 , 62 , 63 ].…”

Section: Introductionmentioning

confidence: 99%

“…A similar concept was a adopted with a more readily available target image domain-cadaver images [ 76 ]. Labeling was expedited for the cadaver endoscopic imaging by using robot kinematic information [ 6 , 58 , 77 , 78 ]. Although both image domains contain some realistic visual details and texture, cadaver data acquisition is expensive.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation

Jiang

Chitrakar

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

Accurate semantic image segmentation from medical imaging can enable intelligent vision-based assistance in robot-assisted minimally invasive surgery. The human body and surgical procedures are highly dynamic. While machine-vision presents a promising approach, sufficiently large training image sets for robust performance are either costly or unavailable. This work examines three novel generative adversarial network (GAN) methods of providing usable synthetic tool images using only surgical background images and a few real tool images. The best of these three novel approaches generates realistic tool textures while preserving local background content by incorporating both a style preservation and a content loss component into the proposed multi-level loss function. The approach is quantitatively evaluated, and results suggest that the synthetically generated training tool images enhance UNet tool segmentation performance. More specifically, with a random set of 100 cadaver and live endoscopic images from the University of Washington Sinus Dataset, the UNet trained with synthetically generated images using the presented method resulted in 35.7% and 30.6% improvement over using purely real images in mean Dice coefficient and Intersection over Union scores, respectively. This study is promising towards the use of more widely available and routine screening endoscopy to preoperatively generate synthetic training tool images for intraoperative UNet tool segmentation.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation

Jiang

Chitrakar

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…In contrast to the broader robotics field, machine learning, especially the deep learning, is not as popular in the surgical robotic research community. This may be due to: 1) the surgical data are often expensive so it is challenging to collect a big amount of data for training deep neural networks [79,80], 2) the known challenging problems, such as environmental perception and dynamic planning, are not solved and can not reach desired reliability in the robotic surgery context [81][82][83][84], 3) the focus of the research community still lies on designing new robots, rather than improving robots' performance based on learning algorithms, 4) comparing with deep learning, the classical modeling methods are easier to interpolate and the performances are easier to predict, thus it is easier to predict the robot reliability.…”

Section: Discussionmentioning

confidence: 99%

The Raven Open Surgical Robotic Platforms: A Review and Prospect

2019

APH

View full text Add to dashboard Cite

The Raven I and the Raven II surgical robots, as open research platforms, have been serving the robotic surgery research community for ten years. The paper 1) briefly presents the Raven I and the Raven II robots, 2) reviews the recent publications that are built upon the Raven robots, aim to be applied to the Raven robots, or are directly compared with the Raven robots, and 3) uses the Raven robots as a case study to discuss the popular research problems in the research community and the trend of robotic surgery study. Instead of being a thorough literature review, this work only reviews the works formally published in the past three years and uses these recent publications to analyze the research interests, the popular open research problems, and opportunities in the topic of robotic surgery.

show abstract

“…The automated and robust segmentation of gastrointestinal (GI) instruments plays a vital role in the success of minimally invasive surgeries [1]. The instrument segmentation aids the doctors or surgical robots to overcome the complexities during performing lower GI endoscopy (colonoscopy) or upper GI endoscopy (gastroscopy) surgeries specifying the precise location, orientation, and current status of the instrument [2]. However, the automated segmentation of GI instruments from the colonoscopy or gastroscopy frames is challenging since the instruments operate in an uncontrolled environment and suffer from illumination changes, mirror reflections.…”

Section: Introductionmentioning

confidence: 99%

EM-Net: An Efficient M-Net for segmentation of surgical instruments in colonoscopy frames

Banik

Roy

Bhattacharjee

2021

NMI

View full text Add to dashboard Cite

This paper addresses the Instrument Segmentation Task, a subtask for the “MedAI: Transparency in Medical Image Segmentation” challenge. To accomplish the subtask, our team “Med_Seg_JU” has proposed a deep learning-based framework, namely “EM-Net: An Efficient M-Net for segmentation of surgical instruments in colonoscopy frames”. The proposed framework is inspired by the M-Net architecture. In this architecture, we have incorporated the EfficientNet B3 module with U-Net as the backbone. Our proposed method obtained a JC of 0.8205, DSC of 0.8632, PRE of 0.8464, REC of 0.9005, F1 of 0.8632, and ACC of 0.9799 as evaluated by the challenge organizers on a separate test dataset. These results justify the efficacy of our proposed method in the segmentation of the surgical instruments.

show abstract

Surgical instrument segmentation for endoscopic vision with data fusion of cnn prediction and kinematic pose

Cited by 48 publications

References 19 publications

Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation

Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation

The Raven Open Surgical Robotic Platforms: A Review and Prospect

EM-Net: An Efficient M-Net for segmentation of surgical instruments in colonoscopy frames

Contact Info

Product

Resources

About