Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking

Lin, Shu; Miao, Albert J.; Lu, Jingpei; Yu, Shunkai; Chiu, Zih-Yun; Richter, Florian; Yip, Michael C.

doi:10.1109/icra48891.2023.10160746

Cited by 12 publications

(3 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It weakens the impact of dynamic feature points brought by surgical instruments on iterative reconstruction. Some researcher uses a semantic perception framework combined with semantic segmentation and geometric information [7]. The core idea is to use a surfel described by five parameters to track the intracavitary tissue.…”

Section: Dynamic Object Detection and Image Segmentationmentioning

confidence: 99%

“…When the processed data amount is large, all four methods of image segmentation face real-time requirements. Due to the presence of RGB-D sensors, performance of [8] and [9] vary under different lighting conditions and scene structures Due to the existence of training network, including the deep learning network, the first two methods ([4] and [7]) of image segmentation and all three solutions to data scarcity lack the ability to generalize in different scenarios.…”

Section: Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

Surgical robot navigation based on SLAM technology

Lai

2024

ACE

View full text Add to dashboard Cite

With the widespread application of surgical robots and the development of computer vision, SLAM-applicated surgery is receiving increasing attention. However, due to the unique surgical environment, SLAM faces some challenges. Two key issues will be discussed in this article: dynamic object detection and image segmentation, as well as scene reconstruction under data scarcity. Firstly, dynamic object detection and image segmentation is an important issue in SLAM applications. During the surgical process, doctors often use surgical instruments, which may partially or completely obscure the object, making it difficult to detect the target. Methods based on traditional feature matching may not be able to accurately detect dynamic targets perform image segmentation. Therefore, this article will combine semantic networks for analysis to improve the performance. In addition, scene reconstruction under data scarcity is another challenge in SLAM applications. Traditional SLAM methods typically rely on a large amount of feature points or map data. But in surgery, due to the complexity of occlusion and geometric structure, reliable data may not be easily obtained. This article will develop with the steps of reconstruction and analyze feasible methods that can improve the accuracy and stability of reconstruction. To conclude, this article will concentrate on these two issues, analyze recent papers, and ultimately summarize some feasible solutions, providing ideas and references for other researchers in this field.

show abstract

Section: Dynamic Object Detection and Image Segmentationmentioning

confidence: 99%

Section: Limitationsmentioning

confidence: 99%

Surgical robot navigation based on SLAM technology

Lai

2024

ACE

View full text Add to dashboard Cite

show abstract

“…Computational efficiency represents a further challenge for real-time perception. A recent trend in this direction is the combination of 3D reconstruction with other surgical computer-vision tasks, such as tool segmentation or tissue tracking, in synergistic (and ideally more efficient) network architectures [38], [39]. Nevertheless, the processing of highresolution raw images to produce 3D point clouds still introduces significant delays.…”

Section: A Image-based Depth Estimationmentioning

confidence: 99%

Dense 3D Reconstruction Through Lidar: A New Perspective on Computer-Integrated Surgery

Caccianiga¹,

Kuchenbecker²

2022

Proceedings of the 14th Hamlyn Symposium on Medical Robotics 2022

View full text Add to dashboard Cite

Technical innovations in sensing and computation are quickly advancing the Ô¨Åeld of computer-integrated surgery. From one side, spectral imaging and biophoton- ics are strongly contributing to intraoperative diagnostics and decision making. Simultaneously, learning-based algorithms are reshaping the concept of assistance and prediction in surgery. In this fast-evolving panorama, we strongly believe there is still a need for robust geometric reconstruction of the surgical Ô¨Åeld whether the goal is traditional surgical assistance or partial or full autonomy. 3D reconstruction in surgery has been investigated almost only in the space of mono and stereoscopic visual imaging because surgeons always view the proce- dure through a clinical endoscope. Compared to using traditional computer vision, deep learning has made signiÔ¨Åcant progress in creating high-quality 3D recon- structions and dense maps from such data streams, especially for monocular simultaneous localization and mapping (SLAM) [1]. The main limitations are linked to reliability, generalization, and computational cost. Meanwhile, lidar (light detection and ranging) has greatly expanded in use, especially in SLAM for robotics, terrestrial vehicles, and drones. Lidar sensors explicitly measure the depth Ô¨Åeld rather than inferring it from camera images. The technology is evolving quickly thanks to the upsurge of mixed and augmented reality in consumer mobile devices [2]: high-resolution, short- range, miniaturized lidar sensors are expected soon. In parallel to these developments, the concept of multiple-viewpoint surgical imaging was proposed in the early 2010‚Äôs in the context of magnetic actuation and micro-invasive surgery [3]. In routine clinical practice, however, the use of multiple trans-abdominal cannulae still limits the kinematics of the camera and instruments to have a Ô¨Åxed pivot point at the body wall. For this reason, here we propose an approach in which each surgical cannula can potentially hold a miniature lidar. We envision that exploring this powerful sensing tech- nology and enabling multi-viewpoint imaging without disrupting the current surgical workÔ¨Çow will yield far more accurate and complete 3D reconstructions of the surgical Ô¨Åeld, which opens new opportunities for the future of computer-integrated surgery.

show abstract