Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery

Pore, Ameya; Corsi, Davide; Marchesini, Enrico; Dall’Alba, Diego; Casals, Alı́cia; Farinelli, Alessandro; Fiorini, Paolo

doi:10.1109/iros51168.2021.9636175

Cited by 27 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Safety Constraints Especially in a surgical context, it is desirable for an RL agent to learn to avoid certain unsafe behaviors as it completes the task (Pore et al, 2021a). Although these undesired behaviors cause no real damage while learning in simulation, they are unacceptable during task execution.…”

Section: Multi-instrument Collaborationmentioning

confidence: 99%

LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

Scheikl¹,

Gyenes²,

Younis³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent advances in reinforcement learning (RL) have increased the promise of introducing cognitive assistance and automation to robot-assisted laparoscopic surgery (RALS). However, progress in algorithms and methods depends on the availability of standardized learning environments that represent skills relevant to RALS. We present LapGym, a framework for building RL environments for RALS that models the challenges posed by surgical tasks, and sofa env, a diverse suite of 12 environments. Motivated by surgical training, these environments are organized into 4 tracks: Spatial Reasoning, Deformable Object Manipulation & Grasping, Dissection, and Thread Manipulation. Each environment is highly parametrizable for increasing difficulty, resulting in a high performance ceiling for new algorithms. We use Proximal Policy Optimization (PPO) to establish a baseline for model-free RL algorithms, investigating the effect of several environment parameters on task difficulty. Finally, we show that many environments and parameter configurations reflect well-known, open problems in RL research, allowing researchers to continue exploring these fundamental problems in a surgical context. We aim to provide a challenging, standard environment suite for further development of RL for RALS, ultimately helping to realize the full potential of cognitive surgical robotics. LapGym is publicly accessible through GitHub (https://github.com/ScheiklP/lap_gym).

show abstract

Section: Multi-instrument Collaborationmentioning

confidence: 99%

LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

Scheikl¹,

Gyenes²,

Younis³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To perform the training of our robot, we rely on Unity3D, a the popular engine originally designed for game development, that has recently been adopted for robotics simulation [42], [50]. In particular, the built-in physics engine, the powerful 3D rendering algorithm and the time control system (which allows to speed up the simulation by more than 10 times), have made Unity3D a very powerful tool in these contexts [29].…”

Section: Appendix B Technical Specifications Of the Robotmentioning

confidence: 99%

Verifying Learning-Based Robotic Navigation Systems

Amir¹,

Corsi²,

Yerushalmi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) has become a dominant deep-learning paradigm for various tasks in which complex policies are learned within reactive systems. In parallel, there has recently been significant research on verifying deep neural networks. However, to date, there has been little work demonstrating the use of modern verification tools on real, DRLcontrolled systems. In this case-study paper, we attempt to begin bridging this gap, and focus on the important task of mapless robotic navigation -a classic robotics problem, in which a robot, usually controlled by a DRL agent, needs to efficiently and safely navigate through an unknown arena towards a desired target. We demonstrate how modern verification engines can be used for effective model selection, i.e., the process of selecting the best available policy for the robot in question from a pool of candidate policies. Specifically, we use verification to detect and rule out policies that may demonstrate suboptimal behavior, such as collisions and infinite loops. We also apply verification to identify models with overly conservative behavior, thus allowing users to choose superior policies that are better at finding an optimal, shorter path to a target. To validate our work, we conducted extensive experiments on an actual robot, and confirmed that the suboptimal policies detected by our method were indeed flawed. We also compared our verification-driven approach to state-of-the-art gradient attacks, and our results demonstrate that gradient-based methods are inadequate in this setting. Our work is the first to demonstrate the use of DNN verification backends for recognizing suboptimal DRL policies in real-world robots, and for filtering out unwanted policies. We believe that the methods presented in this work can be applied to a large range of application domains that incorporate deep-learning-based agents.

show abstract

“…A large amount of work has been done in the application of formal verification to industrial domains. Most of this work focuses on specific domains, such as railway infrastructure management [2,11,22] and medical applications [14,18,23]. More closely related to this paper are works on the modeling and verification of control software.…”

Section: Related Workmentioning

confidence: 99%

Formal verification of an industrial UML-like model using mCRL2 (extended version)

Stramaglia¹,

Keiren²

2022

Preprint

View full text Add to dashboard Cite

Low-code development platforms are gaining popularity. Essentially, such platforms allow to shift from coding to graphical modeling, helping to improve quality and reduce development time. The Cordis SUITE is a low-code development platform that adopts the Unified Modeling Language (UML) to design complex machine-control applications. In this paper we introduce Cordis models and their semantics. To enable formal verification, we define an automatic translation of Cordis models to the process algebraic specification language mCRL2. As a proof of concept, we describe requirements of the control software of an industrial cylinder model developed by Cordis, and show how these can be verified using model checking. We show that our verification approach is effective to uncover subtle issues in the industrial model and its implementation.

show abstract

Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery

Cited by 27 publications

References 16 publications

LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

LapGym -- An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

Verifying Learning-Based Robotic Navigation Systems

Formal verification of an industrial UML-like model using mCRL2 (extended version)

Contact Info

Product

Resources

About