Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario

Long, Yonghao; Cao, Jianfeng; Deguet, Anton; Taylor, Russell H.; Dou, Qi

doi:10.1109/ismr48347.2022.9807505

Cited by 10 publications

(2 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This permits determination of precisely the disorder source and required action utilizing patient individual modeling from deep learning databases. Moreover, the association AR-DT allows an important accuracy in the domain of suturing, tying, and placement contrasted to hand operations [155][156][157][158][159].…”

Section: Discussionmentioning

confidence: 99%

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

Razek

2023

Applied Sciences

View full text Add to dashboard Cite

This contribution is part of the objective of diligent universal care that ensures the well-being of a patient. It aims to analyze and propose enriched image-guided procedures for surgical interventions and restricted delivery of implanted drugs in minimally invasive and non-ionizing circumstances. This analysis is supported by a literature review conducted in two ways. The first aims to illustrate the importance of recent research and applications involved in different topics of the subject; this is mainly the case for the introduction’s literature. The second concerns the literature dedicated to having more detailed information in context; this mainly concerns the citations in the different sections of the article. The universal goals of medical treatments are intended to involve the well-being of the patient and allow medical personnel to test new therapies and carry out therapeutic training without risk to the patient. First, the various functionalities involved in these procedures and the concerns of the magnetic resonance imaging technique (MRI) and ultrasound imaging technique (USI), recent contributions to the subject are reviewed. Second, the intervention procedures guided by the image and the implemented actions are analyzed. Third, the components of the fields involved in MRI are examined. Fourth, the MRI control of the treatments, its performance and its compliance are analyzed. Compatibility with MRI via electromagnetic compatibility (EMC) is conferred and demonstrated for an actuation example. Fifth, the extension of the concepts mentioned in the article, in the context of patient comfort and the training of medical staff is proposed. The main contribution of this article is the identification of the different strategic aids needed in healthcare related to image-assisted robotics, non-ionized, minimally invasive and locally restrictive means. Furthermore, it highlights the benefits of using phantoms based on real biological properties of the body, digital twins under human control, artificial intelligence tools and augmented reality-assisted robotics.

show abstract

Section: Discussionmentioning

confidence: 99%

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

Razek

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…This may be a steppingstone to a natural language robotic platform that partners with the surgeon for a more intuitive and efficient system that enhances patient outcomes. Future iterations of this GPT-robot merging of technology could allow collaboration with surgeons in more profound ways, including in surgical education [1], preplanning surgeries [2], executing specific steps of the surgery [3], and keeping the surgeon informed about any anomalies in the scene or the patient's data [4,5].…”

Section: Introductionmentioning

confidence: 99%

ChatGPT-Enabled daVinci Surgical Robot Prototype: Advancements and Limitations

Pandya

2023

Robotics

View full text Add to dashboard Cite

The daVinci Surgical Robot has revolutionized minimally invasive surgery by enabling greater accuracy and less-invasive procedures. However, the system lacks the advanced features and autonomy necessary for it to function as a true partner. To enhance its usability, we introduce the implementation of a ChatGPT-based natural language robot interface. Overall, our integration of a ChatGPT-enabled daVinci Surgical Robot has potential to expand the utility of the surgical platform by supplying a more accessible interface. Our system can listen to the operator speak and, through the ChatGPT-enabled interface, translate the sentence and context to execute specific commands to alter the robot’s behavior or to activate certain features. For instance, the surgeon could say (even in Spanish) “please track my left tool” and the system will translate the sentence into a specific track command. This specific error-checked command will then be sent to the hardware, which will respond by controlling the camera of the system to continuously adjust and center the left tool in the field of view. We have implemented many commands, including “Find my tools” (tools that are not in the field of view) or start/stop recording, that can be triggered based on a natural conversational context. Here, we present the details of our prototype system, give some accuracy results, and explore its potential implications and limitations. We also discuss how artificial intelligence tools (such as ChatGPT) of the future could be leveraged by robotic surgeons to reduce errors and enhance the efficiency and safety of surgical procedures and even ask for help.

show abstract

Autonomous sequential surgical skills assessment for the peg transfer task in a laparoscopic box-trainer system with three cameras

Fathabadi

Grantner²,

Shebrain³

et al. 2023

Robotica

View full text Add to dashboard Cite

In laparoscopic surgery, surgeons should develop several manual laparoscopic skills before carrying out real operative procedures using a low-cost box trainer. The Fundamentals of Laparoscopic Surgery (FLS) program was developed as a program to assess fundamental knowledge and surgical skills, required for basic laparoscopic surgery. The peg transfer task is a hands-on exam in the FLS program that assists a trainee to understand the relative minimum amount of grasping force necessary to move the pegs from one place to another place without dropping them. In this paper, an autonomous, sequential assessment algorithm based on deep learning, a multi-object detection method, and, several sequential If-Then conditional statements have been developed to monitor each step of a surgeon’s performance. Images from three different cameras are used to assess whether the surgeon executes the peg transfer task correctly and to display a notification on any errors on the monitor immediately. This algorithm improves the performance of a laparoscopic box-trainer system using top, side, and front cameras and removes the need for any human monitoring during a peg transfer task. The developed algorithm can detect each object and its status during a peg transfer task and notifies the resident about the correct or failed outcome. In addition, this system can correctly determine the peg transfer execution time, and the move, carry, and dropped states for each object by the top, side, and front-mounted cameras. Based on the experimental results, the proposed surgical skill assessment system can identify each object at a high score of fidelity, and the train-validation total loss for the single-shot detector (SSD) ResNet50 v1 was about 0.05. Also, the mean average precision (mAP) and Intersection over Union (IoU) of this detection system were 0.741, and 0.75, respectively. This project is a collaborative research effort between the Department of Electrical and Computer Engineering and the Department of Surgery, at Western Michigan University.

show abstract

Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario

Cited by 10 publications

References 63 publications

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

ChatGPT-Enabled daVinci Surgical Robot Prototype: Advancements and Limitations

Autonomous sequential surgical skills assessment for the peg transfer task in a laparoscopic box-trainer system with three cameras

Contact Info

Product

Resources

About