Visual servo control of endoscope-holding robot based on multi-objective optimization: System modeling and instrument tracking

Zhang, Chi; Zhu, Wangru; Peng, Jianqing; Han, Yu; Liu, Wanquan

doi:10.1016/j.measurement.2023.112658

Cited by 8 publications

(6 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These issues could manifest as slow convergence [28] and decreased tracking performance [25]. To overcome these uncertainties, control algorithms such as sliding mode control [29], model predictive control [30], and multi-sensor fusion [31] are deployed in the IBVS framework to improve system robustness. The camera not only provides a view of the lesion but also senses the actual motion of the robot arm, serving as a proprioception mechanism for disturbance compensation.…”

Section: Introductionmentioning

confidence: 99%

Stereo Visual Servoing Control of a Soft Endoscope for Upper Gastrointestinal Endoscopic Submucosal Dissection

Chen,

Wang,

Zhao

et al. 2024

Micromachines

View full text Add to dashboard Cite

Quickly and accurately completing endoscopic submucosal dissection (ESD) operations within narrow lumens is currently challenging because of the environment’s high flexibility, invisible collision, and natural tissue motion. This paper proposes a novel stereo visual servoing control for a dual-segment robotic endoscope (DSRE) for ESD surgery. Departing from conventional monocular-based methods, our DSRE leverages stereoscopic imaging to rapidly extract precise depth data, enabling quicker controller convergence and enhanced surgical accuracy. The system’s dual-segment configuration enables agile maneuverability around lesions, while its compliant structure ensures adaptability within the surgical environment. The implemented stereo visual servo controller uses image features for real-time feedback and dynamically updates gain coefficients, facilitating rapid convergence to the target. In visual servoing experiments, the controller demonstrated strong performance across various tasks. Even when subjected to unknown external forces, the controller maintained robust performance in target tracking. The feasibility and effectiveness of the DSRE were further verified through ex vivo experiments. We posit that this novel system holds significant potential for clinical application in ESD surgeries.

show abstract

Section: Introductionmentioning

confidence: 99%

Stereo Visual Servoing Control of a Soft Endoscope for Upper Gastrointestinal Endoscopic Submucosal Dissection

Chen,

Wang,

Zhao

et al. 2024

Micromachines

View full text Add to dashboard Cite

show abstract

“…Different controllers aim primarily at minimizing misorientation error [ 14 , 15 , 24 ]. Recent studies consider multi-objective approaches or speech-based control as additional variables [ 25 , 26 ]. Null-space projection is commonly used to handle multiple constraints but can fall into local minima, particularly when dealing with inequality constraints like joint states.…”

Section: Introductionmentioning

confidence: 99%

“…Most methods focus on single-tool tracking or employ data-driven methodologies [ 15 , 24 ]. Some even integrate surgeon preferences explicitly or via additional interfaces [ 8 , 25 , 26 ]. However, the extension of these methods for multi-tool handling is still an open research area.…”

Section: Introductionmentioning

confidence: 99%

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery

Fozilov,

Colan,

Davila

et al. 2023

Sensors

View full text Add to dashboard Cite

In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system’s robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system’s potential for real-world application.

show abstract

“…However, it can cause temporary interruption of the surgeon and increase the surgeon's workload for adjusting the endoscope angle. [5] Thus, the active EHS is proposed, [6][7][8][9][10][11][12][13][14] whose endoscope is motorized by machine equipment. Accordingly, various user interfaces have been developed to operate the endoscope instead of manual control, such as head/eye/face movements, [6] voice-based control, [7] finger/footactivated switches, [8] etc.…”

mentioning

confidence: 99%

“…Although these operating methods are convenient, they are less intuitive and efficient than passive EHS. Therefore, the automatic control strategy [9][10][11][12][13][14] based on visual servoing method has been proposed and applied to active EHS to automatically track the tip of surgical instruments, which is a more intuitive and convenient control method.…”

mentioning

confidence: 99%

Design and Optimal Pose‐Constrained Visual Servoing of a Novel Active Flexible Endoscope Holder System for Solo Laparoscopic Surgery

Zhang,

Li,

Huang

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Laparoscopic surgery (LS) has become an effective and widely accepted therapy for patients. However, performing LS requires advanced training and skills, as it demands high precision and dexterity due to visual feedback depending on the collaboration between the surgeon and endoscope assistant, restricted field of view, and limited dexterity inside the body. Thus, a flexible endoscope holder system with an automatic control scheme is proposed to assist LS performance, allowing the surgeon to perform solo LS. A DNA‐inspired helix‐based structure with wide‐angle and constant curvature bending is presented and made of printable nylon with selective laser sintering. The proposed flexible mechanism reduces the cost of manufacturing and assembling. Also, it has a hollow design for embedded sensing and actuation. Then, an optimization method is proposed, which provides a reference for the endoscope installation, robot placement, and pose selection. In addition, aiming at the misorientation problem, a pose‐constrained visual servoing control scheme for the endoscope system is developed. Both simulation and physical experiments are conducted to verify the effectiveness and feasibility of the proposed control scheme. Experimental results demonstrate that the proposed visual servoing scheme can maintain the optimized pose and reduce misorientation during automatic view steering.

show abstract

Visual servo control of endoscope-holding robot based on multi-objective optimization: System modeling and instrument tracking

Cited by 8 publications

References 30 publications

Stereo Visual Servoing Control of a Soft Endoscope for Upper Gastrointestinal Endoscopic Submucosal Dissection

Stereo Visual Servoing Control of a Soft Endoscope for Upper Gastrointestinal Endoscopic Submucosal Dissection

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery

Design and Optimal Pose‐Constrained Visual Servoing of a Novel Active Flexible Endoscope Holder System for Solo Laparoscopic Surgery

Contact Info

Product

Resources

About