Sim-to-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery

Scheikl, Paul Maria; Tagliabue, Eleonora; Gyenes, Balázs; Wagner, Martin; Dall’Alba, Diego; Fiorini, Paolo; Mathis-Ullrich, Franziska

doi:10.1109/lra.2022.3227873

Cited by 26 publications

(8 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The first row represents raw simulation images with a mIoU of 24.73%, FID of 305.00, KID of 0.3739 ± 0.0041, and LPIPS of 0.5820. The second and third rows, attributed to the method presented in [ 28 ], demonstrate improvements in performance. For the random style, the mIoU and LPIPS increase to 45.28% and 0.5834, respectively, while the FID decreases to 110.92 and the KID to 0.1243 ± 0.0035.…”

Section: Resultsmentioning

confidence: 99%

Minimal data requirement for realistic endoscopic image generation with Stable Diffusion

Kaleta,

Dall’Alba,

Płotka

et al. 2023

Int J CARS

Self Cite

View full text Add to dashboard Cite

Purpose Computer-assisted surgical systems provide support information to the surgeon, which can improve the execution and overall outcome of the procedure. These systems are based on deep learning models that are trained on complex and challenging-to-annotate data. Generating synthetic data can overcome these limitations, but it is necessary to reduce the domain gap between real and synthetic data. Methods We propose a method for image-to-image translation based on a Stable Diffusion model, which generates realistic images starting from synthetic data. Compared to previous works, the proposed method is better suited for clinical application as it requires a much smaller amount of input data and allows finer control over the generation of details by introducing different variants of supporting control networks. Results The proposed method is applied in the context of laparoscopic cholecystectomy, using synthetic and real data from public datasets. It achieves a mean Intersection over Union of 69.76%, significantly improving the baseline results (69.76 vs. 42.21%). Conclusions The proposed method for translating synthetic images into images with realistic characteristics will enable the training of deep learning methods that can generalize optimally to real-world contexts, thereby improving computer-assisted intervention guidance systems.

show abstract

Section: Resultsmentioning

confidence: 99%

Minimal data requirement for realistic endoscopic image generation with Stable Diffusion

Kaleta,

Dall’Alba,

Płotka

et al. 2023

Int J CARS

Self Cite

View full text Add to dashboard Cite

show abstract

“…To address this issue, simulation-based training has been employed in previous studies [29], [30]. Nevertheless, the significant differences between simulated environments and real-world scenarios limit the applicability of simulationtrained models [8]. Furthermore, in RAMIS, variations in the mechanical properties and appearance of soft tissues render nonadaptive RL or IL models unsuitable for many surgical settings.…”

Section: A Related Workmentioning

confidence: 99%

“…This creates a need for robust autonomous control systems that can operate effectively in a surgical environment. Several research efforts are currently dedicated to autonomous robotic manipulation of soft objects in RAMIS, such as autonomous fat retraction [8] and intestinal anastomosis using threads manipulation [9]. However, research on increasing computer assistance when manipulating human soft tissues has not yet been explored.…”

mentioning

confidence: 99%

Occlusion-Robust Autonomous Robotic Manipulation of Human Soft Tissues With 3-D Surface Feedback

Hu,

Jones,

Dogar

et al. 2024

IEEE Trans. Robot.

View full text Add to dashboard Cite

Robotic manipulation of 3D soft objects remains challenging in the industrial and medical fields. Various methods based on mechanical modelling, data-driven approaches or explicit feature tracking have been proposed. A unifying disadvantage of these methods is the high computational cost of simultaneous imaging processing, identification of mechanical properties, and motion planning, leading to a need for less computationally intensive methods. We propose a method for autonomous robotic manipulation with 3D surface feedback to solve these issues. First, we produce a deformation model of the manipulated object, which estimates the robots' movements by monitoring the displacement of surface points surrounding the manipulators. Then, we develop a 6-degree-of-freedom velocity controller to manipulate the grasped object to achieve a desired shape. We validate our approach through comparative simulations with existing methods and experiments using phantom and cadaveric soft tissues with the da Vinci Research Kit. The results demonstrate the robustness of the technique to occlusions and various materials. Compared to state-of-the-art linear and data-driven methods, our approach is more precise by 46.5% and 15.9% and saves 55.2% and 25.7% manipulation time, respectively.Index Terms-Shape control, dual arm manipulation, robotic manipulation of soft objects I. INTRODUCTIONR OBOTIC manipulation of soft, deformable objects is a common practice in industry, from fabric folding to food packaging [1], [2], [3], [4], [5], with similar problems observed in medical scenarios. Despite recent progress in the robotic manipulation of deformable linear and planar objects, such as ropes and clothing items, the shape control of 3D objects is still a challenge.In robotic-assisted minimally invasive surgery (RAMIS) [6], soft tissues within the abdomen are manipulated with the teleoperated end-effectors of a surgical robot [7]. Recently, the research focus in this area has trended towards

show abstract

“…This balance is especially precarious if desired and undesired states appear very similar in observation space, for example in PrecisionCuttingEnv. One possible solution to this challenge is curriculum learning Scheikl et al (2023), where punishments for safety violations are incrementally increased throughout training, without curtailing initial exploration.…”

Section: Multi-instrument Collaborationmentioning

confidence: 99%

LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

Scheikl¹,

Gyenes²,

Younis³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent advances in reinforcement learning (RL) have increased the promise of introducing cognitive assistance and automation to robot-assisted laparoscopic surgery (RALS). However, progress in algorithms and methods depends on the availability of standardized learning environments that represent skills relevant to RALS. We present LapGym, a framework for building RL environments for RALS that models the challenges posed by surgical tasks, and sofa env, a diverse suite of 12 environments. Motivated by surgical training, these environments are organized into 4 tracks: Spatial Reasoning, Deformable Object Manipulation & Grasping, Dissection, and Thread Manipulation. Each environment is highly parametrizable for increasing difficulty, resulting in a high performance ceiling for new algorithms. We use Proximal Policy Optimization (PPO) to establish a baseline for model-free RL algorithms, investigating the effect of several environment parameters on task difficulty. Finally, we show that many environments and parameter configurations reflect well-known, open problems in RL research, allowing researchers to continue exploring these fundamental problems in a surgical context. We aim to provide a challenging, standard environment suite for further development of RL for RALS, ultimately helping to realize the full potential of cognitive surgical robotics. LapGym is publicly accessible through GitHub (https://github.com/ScheiklP/lap_gym).

show abstract

Sim-to-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery

Cited by 26 publications

References 25 publications

Minimal data requirement for realistic endoscopic image generation with Stable Diffusion

Minimal data requirement for realistic endoscopic image generation with Stable Diffusion

Occlusion-Robust Autonomous Robotic Manipulation of Human Soft Tissues With 3-D Surface Feedback

LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

Contact Info

Product

Resources

About