Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study

Pore, Ameya; Finocchiaro, Martina; Dall’Alba, Diego; Hernansanz, Albert; Ciuti, Gastone; Arezzo, Alberto; Menciassi, Arianna; Casals, Alı́cia; Fiorini, Paolo

doi:10.1109/iros47612.2022.9981480

Cited by 10 publications

(8 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This first level of robotic assistance is called "intelligent teleoperation" for the specific application of robotic colonoscopy [44]. In this case, the operator guides the tip of the robotic endoscope, while the control system generates the required low level control strategy to enable the execution of the desired motion [44], [45]. Taking as an example the magnetic colonoscopes, the teleoperator guides the colonoscope tool tip and the robotic control system computes the movements of the external permanent magnet to have the robotic capsule reaching the point desired by the operator [44].…”

Section: Case Studymentioning

confidence: 99%

A Framework for the Evaluation of Human Machine Interfaces of Robot-Assisted Colonoscopy

Finocchiaro,

Banfi,

Donaire

et al. 2024

IEEE Trans. Biomed. Eng.

Self Cite

View full text Add to dashboard Cite

The Human Machine Interface (HMI) of intraluminal robots has a crucial impact on the clinician's performance. It increases or decreases the difficulty of the tasks, and is connected to the users' physical and mental stress. Objective: This paper presents a framework to compare and evaluate different HMI for robotic colonoscopy, with the objective of identifying the optimal HMI that minimises the clinician's effort and maximises the clinical outcomes. Methods: The framework comprises a (1) a virtual simulator (clinically validated), (2) wearable sensors measuring the cognitive load, (3) a data collection unit of metrics correlated to the clinical performances, and (4) questionnaires exploring the users' impressions and perceived stress. The framework was tested with 42 clinicians investigating the optimal device for teleoperated control of robotic colonoscopes. Two control devices were selected and compared: a haptic serial-kinematic device and a standard videogame joypad. Results: The haptic device was preferred by the endoscopists, but the joypad enabled better clinical performance and reduced cognitive and physical load. Conclusion: The framework can be used to evaluate different aspects of HMI, both hardware and software, and determine the optimal HMI that can reduce the burden on clinicians while improving the clinical outcome. Significance: The findings of this study, and of future studies performed with this framework, can inform the design and development of HMI for intraluminal robots, leading to improved clinical performance, reduced physical and mental stress for clinicians, and ultimately better patient outcomes.

show abstract

Section: Case Studymentioning

confidence: 99%

A Framework for the Evaluation of Human Machine Interfaces of Robot-Assisted Colonoscopy

Finocchiaro,

Banfi,

Donaire

et al. 2024

IEEE Trans. Biomed. Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…On the other hand, automatic controlling methods have been developed to achieve a higher coverage ratio compared to manual controlling. [10][11][12][13][14][15][16] However, previous research on automatic controllers has limitations. [10][11][12] are more orientated toward assisting the navigation of the capsule, rather than providing an end-to-end closed-loop controller.…”

Section: Introductionmentioning

confidence: 99%

“…13 utilizes multiple infrared cameras to collect information, which may be challenging for clinical setup. 16 requires collections of physician-supervised trajectories in patients, which are hard to collect and may lead to human bias. 15 only allows for translational motions of the magnetic head, without considering rotation actions.…”

Section: Introductionmentioning

confidence: 99%

Simulation of manual and automatic navigation of magnetically controlled wireless capsule endoscopy examination of human gastric

Li,

Gan,

Duan

et al. 2024

Endoscopic Microscopy XIX

View full text Add to dashboard Cite

The magnetically controlled capsule endoscopy (MCCE) is an emerging modality for assessing gastrointestinal disorders due to its advantages. However, current assignments of MCCE rely on manual controlling and gastric landmarks, which are prone to omissions. We improve the scanning protocol of the MCCE in human gastric using both manual and automatic controlling methods. We design a quantitative scanning coverage ratio to measure the process of MCCE scanning within the human gastric. The proposed scanning coverage ratio is capable of guiding the manual and automatic scanning process of human gastric. Moreover, we design a deep reinforcement learning (DRL) controller for automatically navigating the capsule. Our DRL controller achieves a higher coverage ratio compared to previous research.

show abstract

“…Robot assistance (Intelligent teleoperation [15]) -The operator controls directly the final position of the colonoscope using a master console, benefiting of a simplified and smart control strategy. In the case of magnetic endoscopes, the robotic system will autonomously compute the movements of the external permanent magnet to have the robotic capsule reaching the point desired by the operator [15,26]. Therefore, the control would be more intuitive and easy for the user.…”

Section: Levels Of Autonomy In Robotic Colonoscopymentioning

confidence: 99%

“…[15]) -the colonoscope is steered autonomously trough the lumen, thanks to computer vision techniques [27] or other methods [36]. The control strategy generated by the robotic system could be overridden by the operator which needs to constantly supervise the motion of the robot and approve its decision [15,26,37].…”

mentioning

confidence: 99%

Automatic hands-free visualization of a six degrees of freedom agent within a complex anatomical space

Finocchiaro

View full text Add to dashboard Cite

(English) Over the years, a continuous development of intraluminal procedures resulted in strong benefits for the patients. Reduced blood loss, lower risk of infections, diminished scaring impact and quicker recovery time are among the most valuable ones. However, these improvements imposed high mental and physical stress to the clinicians. In this context, the introduction of robotic technologies has resulted in notable improvements in terms of flexibility of the endoscopes and control stability, by designing multi-steerable snake like robots and endoscopic capsules. Robotic devices also introduced additional degrees of freedom (DOF) to control, as well as sensing information to process, posing the basis for a new framework of human-robot interaction. Therefore, in the context of intraluminal robotic surgery, the present research focuses on the human-robot interaction, aiming at investigating the optimal way to design Human Machine Interface (HMI) with multiple levels of assistance. Accordingly, a modular bio-engineering framework was designed and developed for the analysis, evaluation and comparison of different HMI for robot assisted endoluminal procedure (i.e., colonoscopy). The main component of the framework is a virtual simulator of the robotic colonoscopy procedure, developed using the SOFA. The simulator, endowed with 3D models of colons reconstructed from real patients' CT scans, realistically reproduces the anatomy and its performance during the robotic medical procedure in terms of timings, visual rendering and mechanical behaviour. Its open design allows to measure several metrics correlated with the quality of the procedure (e.g., force exerted on the intestinal walls, timings etc.) and of control (e.g., smoothness of trajectory). Therefore, the different HMI can be used to control the robotic endoscope in the virtual simulator and tested with user studies involving the endoscopists. This framework also comprises the use of wearable sensors to measure the cognitive load of the users through physiological data when testing the HMI in the simulation environment. Finally, a set of questionnaires were designed to be filled by the subjects after the tests for measuring their perceived physical and mental stress, and their overall impression on the interfaces. The framework was tested for the first time by 42 clinicians with the goal of deriving the optimal device for teleoperated control of robotic colonoscopes. To this end, a preliminary survey was driven among 71 endoscopists to derive the main characteristics and configuration of the control device desired by the final users. Accordingly, two selected systems were compared with the framework: an haptic serial-kinematic device and a standard videogame joypad. This users' test represented a first case study for the validation of the framework allowing to compare different HMI and derive their optimal features. Nevertheless, being the framework highly modular and open, is meant to be applied for the testing of different aspects of the HMI, both software and hardware e.g., types of feedback, control strategies etc. Indeed, the final goal of the framework, and more in general of the present thesis, is to extract insights, guidelines and metrics over the design of the next generation intraluminal robotic devices. (Català) Al llarg dels anys, un desenvolupament continu dels procediments intraluminals va donar lloc a grans beneficis per als pacients. La reducció de la pèrdua de sang, el menor risc d'infecció, la disminució de l'impacte espantant i el temps de recuperació més ràpid es troben entre els més valuosos. Tanmateix, aquestes millores van imposar un alt estrès mental i físic als metges. En aquest context, la introducció de les tecnologies robòtiques ha donat com a resultat millores notables pel que fa a la flexibilitat dels endoscopis i l'estabilitat del control, mitjançant el disseny de serps multidireccionables com robots i càpsules endoscòpiques. No obstant això, també va introduir graus de llibertat addicionals (DOF) per controlar, com a informació de detecció a processar, posant la base per a un nou marc d'interacció humà-robot. Per tant, en el context de la cirurgia robòtica intraluminal, la present investigació se centra en la interacció home-robot, amb l'objectiu d'investigar i dissenyar una interfície humà-màquina (HMI) d'autonomia multinivell. En conseqüència, es va dissenyar i desenvolupar un marc de bioenginyeria modular per a l'anàlisi, l'avaluació i la comparació de l'HMI per al procediment endoluminal assistit per robot (és a dir, la colonoscòpia). El component principal del marc és un simulador virtual del procediment de colonoscòpia, desenvolupat mitjançant el SOFA. El simulador, dotat de models 3D de còlons reconstruïts a partir de TAC de pacients reals, reprodueix de manera realista el procediment mèdic en termes de temps, representació visual i comportament mecànic. El seu disseny obert permet mesurar diverses mètriques correlacionades amb la qualitat del procediment (per exemple, força exercida sobre les parets intestinals, temps, etc.) i de control (per exemple, suavitat de la trajectòria). Per tant, les diferents HMI es poden connectar o implementar en el simulador virtual, i provar-se amb estudis d'usuaris que involucren els endoscopistes. El marc també inclou l'ús de sensors portàtils per mesurar la càrrega cognitiva dels usuaris mitjançant dades fisiològiques quan es prova l'HMI en l'entorn de simulació. Finalment, es dissenya un conjunt de qüestionaris per ser administrats als subjectes després de les proves per fer el seguiment de l'estrès físic i mental percebut i la seva impressió global a les interfícies. El marc es va provar per primera vegada amb 42 metges que investigaven el dispositiu òptim per al control teleoperat de colonoscopis robòtics. Amb aquesta finalitat, es va dur a terme una enquesta preliminar entre 71 endoscopistes per derivar les principals característiques i configuració del dispositiu de control desitjat pels usuaris finals. En conseqüència, es van comparar dos sistemes seleccionats amb el marc: un dispositiu cinemàtic en sèrie hàptic i un joypad estàndard de videojocs. Aquesta prova d'usuaris va representar un primer cas pràctic per a la validació del framework que permet comparar diferents HMI i derivar les característiques òptimes. No obstant això, al ser el framework altament modular i obert, està pensat per ser aplicat per a la prova de diferents aspectes de l'HMI, tant de programari com de maquinari, p. tipus de retroalimentació, estratègies de control, etc. De fet, l'objectiu final del marc, i més en general de la present tesi, és extreure coneixements, directrius i mètriques sobre el disseny dels dispositius robòtics intraluminals de nova generació.

show abstract

Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study

Cited by 10 publications

References 19 publications

A Framework for the Evaluation of Human Machine Interfaces of Robot-Assisted Colonoscopy

A Framework for the Evaluation of Human Machine Interfaces of Robot-Assisted Colonoscopy

Simulation of manual and automatic navigation of magnetically controlled wireless capsule endoscopy examination of human gastric

Automatic hands-free visualization of a six degrees of freedom agent within a complex anatomical space

Contact Info

Product

Resources

About