Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning

Scheikl, Paul Maria; Gyenes, Balázs; Davitashvili, Tornike; Younis, Rayan; Schulze, A.; Müller‐Stich, Beat P.; Neumann, Gerhard; Wagner, Martin; Mathis-Ullrich, Franziska

doi:10.1109/iros51168.2021.9636193

Cited by 16 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, learningbased approaches, representatively RL, enable robots to flexibly learn to perform tasks from collected data. These methods thus do not require task-specific control strategies and have shown improved generalization capabilities in automating complex surgical tasks [14][15][16][17][18][19][20]. Promising as it is, running existing RL methods in surgical tasks is still inefficient due to the exploration challenge, which is often alleviated through substantial reward engineering for each task.…”

Section: Related Workmentioning

confidence: 99%

“…Automating surgical tasks has been increasingly desired to improve surgical efficiency, with many efforts being made on different tasks such as suturing [2][3][4], endoscope control [5][6][7], tissue manipulation [8][9][10] and pattern cutting [11][12][13]. Recently, reinforcement learning (RL) approaches have exhibited high scalability to learn diverse control policies and yielded promising performance in surgical automation [14][15][16][17][18][19][20], but typically require extensive data collection to solve a task if no prior knowledge is given. This issue, known as the exploration challenge, gives rise to the idea of providing expert knowledge from demonstration data to an RL agent [21].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot

Huang¹,

Chen²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue is known as the exploration challenge, which can be alleviated by providing expert demonstrations to an RL agent. Yet, how to make effective use of demonstration data to improve exploration efficiency still remains an open challenge. In this work, we introduce Demonstration-guided EXploration (DEX), an efficient reinforcement learning algorithm that aims to overcome the exploration problem with expert demonstrations for surgical automation. To effectively exploit demonstrations, our method estimates expert-like behaviors with higher values to facilitate productive interactions, and adopts non-parametric regression to enable such guidance at states unobserved in demonstration data. Extensive experiments on 10 surgical manipulation tasks from SurRoL, a comprehensive surgical simulation platform, demonstrate significant improvements in the exploration efficiency and task success rates of our method. Moreover, we also deploy the learned policies to the da Vinci Research Kit (dVRK) platform to show the effectiveness on the real robot. Code is available at https://github.com/med-air/DEX.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot

Huang¹,

Chen²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent advancements in reinforcement learning (RL) have extended the capacity of neural networks into the physical domain, allowing for intelligent control of robotic agents in a dynamic environment. Accordingly, RL in robotics has the potential to advance key industries, from manufacturing [1] to healthcare [2], agriculture [3] or transportation [4].…”

Section: Introductionmentioning

confidence: 99%

Renaissance Robot: Optimal Transport Policy Fusion for Learning Diverse Skills

Tan¹,

Senanayake²,

Ramos³

2022

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (RL) is a promising approach to solving complex robotics problems. However, the process of learning through trial-and-error interactions is often highly time-consuming, despite recent advancements in RL algorithms. Additionally, the success of RL is critically dependent on how well the reward-shaping function suits the task, which is also time-consuming to design. As agents trained on a variety of robotics problems continue to proliferate, the ability to reuse their valuable learning for new domains becomes increasingly significant. In this paper, we propose a post-hoc technique for policy fusion using Optimal Transport theory as a robust means of consolidating the knowledge of multiple agents that have been trained on distinct scenarios. We further demonstrate that this provides an improved weights initialisation of the neural network policy for learning new tasks, requiring less time and computational resources than either retraining the parent policies or training a new policy from scratch. Ultimately, our results on diverse agents commonly used in deep RL show that specialised knowledge can be unified into a "Renaissance agent", allowing for quicker learning of new skills.

show abstract

“…Robot-assisted surgery has transformed the field of minimally invasive surgery by allowing surgeons to operate with greater dexterity and precision and improving patient outcomes. Characterizing the interactions among the surgical instruments and important objects and anatomical structures within the surgical scene can provide context awareness [1], which is crucial for various downstream tasks, such as cognitive assistance [2], skill evaluation [3], [4], [5], [6] and error detection [7], [8], [9], [10], [11].…”

Section: Introductionmentioning

confidence: 99%

Local Context Embedding Neural Network for Scene Semantic Segmentation

Dai

Ding

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Surgical context inference has recently garnered significant attention in robot-assisted surgery as it can facilitate workflow analysis, skill assessment, and error detection. However, runtime context inference is challenging since it requires timely and accurate detection of the interactions among the tools and objects in the surgical scene based on the segmentation of video data. On the other hand, existing stateof-the-art video segmentation methods are often biased against infrequent classes and fail to provide temporal consistency for segmented masks. This can negatively impact the context inference and accurate detection of critical states. In this study, we propose a solution to these challenges using a Space-Time Correspondence Network (STCN). STCN is a memory network that performs binary segmentation and minimizes the effects of class imbalance. The use of a memory bank in STCN allows for the utilization of past image and segmentation information, thereby ensuring consistency of the masks. Our experiments using the publicly-available JIGSAWS dataset demonstrate that STCN achieves superior segmentation performance for objects that are difficult to segment, such as needle and thread, and improves context inference compared to the state-of-the-art. We also demonstrate that segmentation and context inference can be performed at runtime without compromising performance.

show abstract

Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning

Cited by 16 publications

References 10 publications

Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot

Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot

Renaissance Robot: Optimal Transport Policy Fusion for Learning Diverse Skills

Local Context Embedding Neural Network for Scene Semantic Segmentation

Contact Info

Product

Resources

About