Autonomous and stable tracking of endoscope instrument tools with monocular camera

Song, Kai‐Tai; Chen, Chun-Ju

doi:10.1109/aim.2012.6266023

Cited by 14 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Song and Chen (2012) , authors proposed to use a monocular webcam mounted on a robotic pan-tilt platform to track two laparoscopic instruments with two colored rings attached to each instrument. They employed the estimated 2D image coordinates of the fiducial markers to control all the degrees of freedom of the robotic platform.…”

Section: Related Workmentioning

confidence: 99%

Robotic Endoscope Control Via Autonomous Instrument Tracking

Gruijthuijsen

García-Peraza-Herrera

Borghesan

et al. 2022

Front. Robot. AI

View full text Add to dashboard Cite

Many keyhole interventions rely on bi-manual handling of surgical instruments, forcing the main surgeon to rely on a second surgeon to act as a camera assistant. In addition to the burden of excessively involving surgical staff, this may lead to reduced image stability, increased task completion time and sometimes errors due to the monotony of the task. Robotic endoscope holders, controlled by a set of basic instructions, have been proposed as an alternative, but their unnatural handling may increase the cognitive load of the (solo) surgeon, which hinders their clinical acceptance. More seamless integration in the surgical workflow would be achieved if robotic endoscope holders collaborated with the operating surgeon via semantically rich instructions that closely resemble instructions that would otherwise be issued to a human camera assistant, such as “focus on my right-hand instrument.” As a proof of concept, this paper presents a novel system that paves the way towards a synergistic interaction between surgeons and robotic endoscope holders. The proposed platform allows the surgeon to perform a bimanual coordination and navigation task, while a robotic arm autonomously performs the endoscope positioning tasks. Within our system, we propose a novel tooltip localization method based on surgical tool segmentation and a novel visual servoing approach that ensures smooth and appropriate motion of the endoscope camera. We validate our vision pipeline and run a user study of this system. The clinical relevance of the study is ensured through the use of a laparoscopic exercise validated by the European Academy of Gynaecological Surgery which involves bi-manual coordination and navigation. Successful application of our proposed system provides a promising starting point towards broader clinical adoption of robotic endoscope holders.

show abstract

Section: Related Workmentioning

confidence: 99%

Robotic Endoscope Control Via Autonomous Instrument Tracking

Gruijthuijsen

García-Peraza-Herrera

Borghesan

et al. 2022

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…It is robust to illumination variations, thanks to the probability-based technique. In authors' previous work [5], an image tracking system was developed to provide a stable camera view of endoscopic surgery.…”

Section: Introductionmentioning

confidence: 99%

Image tracking of laparoscopic instrument using spiking neural networks

Chen

Huang

Song

2013

2013 13th International Conference on Control, Automation and Systems (ICCAS 2013)

Self Cite

View full text Add to dashboard Cite

Minimally Invasive Surgery (MIS) has become more and more popular in recent years. An endoscopic image tracking system will assist surgeons to adjust the field of view autonomously in MIS. In this paper, we propose a novel image tracking algorithm based on natural features of surgical instruments. We suggest to use texture and geometric features in laparoscopic instrument imagery and to adopt a spiking neural network approach for object detection; considering color will be affected by lighting and the white balance conditions in the endoscope imagery. To enhance tracking performance, we further design a Kalman filter to combine with the neuro-based tracker. The instrument can be detected more robustly despite of deformation of the instrument image during surgery. A laparoscopic video has been tested to verify the developed methods. Experimental results show that two instrnments can be distinguished and tracked simultaneously in the surgical video.

show abstract

“…Current vision based robotic controlled laparoscopic systems [5]- [13] work by localizing the instrument position in 2D, planning a path and moving the robot. For controlling the depth, the geometrical relations between the instrument [13] or the relation between the visible tool/tools and the size of the whole scene [11], [12] are utilized. Although the point may be defined by a tool but this can cause problems; first the depth can be hard to estimate accurately, secondly the end position of the laparoscope may not have the desired field of view so this approach to navigation is less intuitive.…”

Section: Introductionmentioning

confidence: 99%

Instrument State Recognition and Tracking for Effective Control of Robotized Laparoscopic Systems

Sahu¹,

Moerman²,

Mewes³

et al. 2016

IJMERR

View full text Add to dashboard Cite

Surgical robots are an important component for delivering advanced paradigm shifting technology such as image guided surgery and navigation. However, for robotic systems to be readily adopted into the operating room they must be easy and convenient to control and facilitate a smooth surgical workflow. In minimally invasive surgery, the laparoscope may be held by a robot but controlling and moving the laparoscope remains challenging. It is disruptive to the workflow for the surgeon to put down the tools to move the robot in particular for solo surgery approaches. This paper proposes a novel approach for naturally controlling the robot mounted laparoscope's position by detecting a surgical grasping tool and recognizing if its state is open or close. This approach does not require markers or fiducials and uses a machine learning framework for tool and state recognition which exploits naturally occurring visual cues. Furthermore a virtual user interface on the laparoscopic image is proposed that uses the surgical tool as a pointing device to overcome common problems in depth perception. Instrument detection and state recognition are evaluated on in-vivo and ex-vivo porcine datasets. To demonstrate the practical surgical application and real time performance the system is validated in a simulated surgical environment. 

show abstract

Autonomous and stable tracking of endoscope instrument tools with monocular camera

Cited by 14 publications

References 12 publications

Robotic Endoscope Control Via Autonomous Instrument Tracking

Robotic Endoscope Control Via Autonomous Instrument Tracking

Image tracking of laparoscopic instrument using spiking neural networks

Instrument State Recognition and Tracking for Effective Control of Robotized Laparoscopic Systems

Contact Info

Product

Resources

About