Hand gesture guided robot-assisted surgery based on a direct augmented reality interface

Wen, Rong; Tay, Wei-Liang; Nguyen, Binh P.; Chng, Chin-Boon; Chui, Chee-Kong

doi:10.1016/j.cmpb.2013.12.018

Cited by 101 publications

(42 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Our system can also be extended to include the control and manipulation of a robot for RF needle insertion. In [22], we described a natural gestures control method for robust interaction with the AR and the surgical robot system. The front camera of the table computer can be used for recognition of hand gestures of surgeons so that strictly sterile interaction can be realized.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery

2017

Self Cite

View full text Add to dashboard Cite

Image-guided surgical procedures are challenged by mono image modality, twodimensional anatomical guidance and non-intuitive human-machine interaction. The introduction of Tablet-based augmented reality (AR) into surgical robots may assist surgeons with overcoming these problems. In this paper, we proposed and developed a robot-assisted surgical system with interactive surgical guidance using tablet-based AR with a Kinect sensor for three-dimensional (3D) localization of patient anatomical structures and intraoperative 3D surgical tool navigation. Depth data acquired from the Kinect sensor was visualized in cone-shaped layers for 3D AR-assisted navigation. Virtual visual cues generated by the tablet were overlaid on the images of the surgical field for spatial reference. We evaluated the proposed system and the experimental results showed that the tablet-based visual guidance system could assist surgeons in locating internal organs, with errors between 1.74 and 2.96 mm. We also demonstrated that the system was able to provide mobile augmented guidance and interaction for surgical tool navigation.

show abstract

Section: Discussionmentioning

confidence: 99%

“…The built-in forward camera of the tablet can also provide an alternative for touchless gesture recognition as in [22] for AR operation if sterility is strictly required.…”

Section: Interactive Surgical Guidancementioning

confidence: 99%

Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery

2017

Self Cite

View full text Add to dashboard Cite

show abstract

“…The actions to test the collaborative robot take into account the various robotic actions covered by the literature [20], among which are the guidance of the laparoscopic camera for the safe movement of the endoscope [21] or a needle insertion [22], the prediction of the end point [23,24], the knotting and unknotting on suture procedures [25], or grasping and lifting on tissue retraction [26]. Ultimately, we selected three actions to be performed by the collaborative robot: center the image from the endoscope, indicate a place to suture, and stretch the thread to suture.…”

Section: Algorithm For Movement Detectionmentioning

confidence: 99%

Dynamic Gesture Recognition Using a Smart Glove in Hand-Assisted Laparoscopic Surgery

et al. 2018

View full text Add to dashboard Cite

This paper presents a methodology for movement recognition in hand-assisted laparoscopic surgery using a textile-based sensing glove. The aim is to recognize the commands given by the surgeon's hand inside the patient's abdominal cavity in order to guide a collaborative robot. The glove, which incorporates piezoresistive sensors, continuously captures the degree of flexion of the surgeon's fingers. These data are analyzed throughout the surgical operation using an algorithm that detects and recognizes some defined movements as commands for the collaborative robot. However, hand movement recognition is not an easy task, because of the high variability in the motion patterns of different people and situations. The data detected by the sensing glove are analyzed using the following methodology. First, the patterns of the different selected movements are defined. Then, the parameters of the movements for each person are extracted. The parameters concerning bending speed and execution time of the movements are modeled in a prephase, in which all of the necessary information is extracted for subsequent detection during the execution of the motion. The results obtained with 10 different volunteers show a high degree of precision and recall.

show abstract

“…Los primeros usos de este dispositivo para tareas ajenas a su concepción inicial de instrumento para video juegos, se dieron en sistemas de reconocimiento de gestos de manera general ( [15], [16], [17], [18]). Posteriormente se empezó a utilizar el dispositivo en diversas aplicaciones, entre ellas la replicación de movimientos en robots ( [19], [20]), la rehabilitación de pacientes [21], [22] y, finalmente, su uso como interfaz natural para el control de dispositivos quirúrgicos, como el caso del uso del Kinect para reconocer un protocolo de gestos en la inserción de guías dentro del paciente [23]; la evaluación del mismo dispositivo como generador de movimientos en un simulador quirúrgico [24], o el uso del Kinect para el reconocimiento de un protocolo de gestos en una biopsia cerebral [25]. El mayor aporte de este proyecto es la utilización del Kinect para mover dos robots quirúrgicos en operaciones de laparoscopia.…”

Section: Dispositivo Kinect Y Su Interacción Con Robosurgeryunclassified

Sistema de captura de gestos con KINECT para la manipulación de robots quirúrgicos virtuales

Hurtado-Chaves¹,

Nástar-Guacales²,

Vivas-Albán³

2015

Iteckne

View full text Add to dashboard Cite

Resumen-Este artículo presenta la aplicación de un sistema de captura de gestos con el fin de manipular dos robots quirúr-gicos virtuales. El sistema de captura de gestos se realiza por medio de un dispositivo Kinect, el cual detecta el movimiento de las manos del usuario con el fin de mover los robots quirúr-gicos, y de su rodilla derecha, utilizada para cambiar los instrumentos quirúrgicos de cada robot. Este sistema de captura con interfaz natural se prueba en un simulador quirúrgico para operaciones de laparoscopia, el cual consta de un robot porta endoscopio que se manipula con un joystick, y los dos robots quirúrgicos cuyos órganos terminales son guiados a partir de las señales captadas por el dispositivo Kinect, las cuales son transformadas en movimientos articulares de los robots debido al modelo cinemático. El sistema se prueba en una rutina de extracción de vesícula (colecistectomía) robotizada, permitiendo al usuario un mejor manejo gracias al uso de esta interfaz natural.Palabras clave-Dispositivo Kinect, interfaces naturales, robots quirúrgicos, simuladores quirúrgicos.Abstract-This article presents a gesture capture system for manipulating two virtual surgical robots. Gesture capture uses Kinect device to detect the movement of the user´s hand to move the surgical robots, and his right knee to change the needed surgical tools. This natural interface capture system is proved on a surgical simulator for laparoscopic, that has a holder endoscopic robot manipulated by a joystick, and two surgical robots that are guided by the Kinect device. Kinematics model is used to transform Kinect captured signals to articular movements for the robots. System is tested on a robotic cholecystectomy, allowing a more natural manipulation to the user.

show abstract

Hand gesture guided robot-assisted surgery based on a direct augmented reality interface

Cited by 101 publications

References 13 publications

Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery

Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery

Dynamic Gesture Recognition Using a Smart Glove in Hand-Assisted Laparoscopic Surgery

Sistema de captura de gestos con KINECT para la manipulación de robots quirúrgicos virtuales

Contact Info

Product

Resources

About