Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigation

Marchesini, Enrico; Corsi, Davide; Farinelli, Alessandro

doi:10.1109/iros51168.2021.9635925

Cited by 20 publications

(12 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We will need two learning rates, one for the actor-network (α actor ) and the other for the critic network (α critic ) because we have two types of networks. The use of the percent of times that a random action is executed is explained by equation (5).…”

Section: Ddpg + Her and Gamentioning

confidence: 99%

“…Reinforcement Learning (RL) [1] has recently been applied to a variety of applications, including robotic table tennis [2], surgical robot planning [3], rapid motion planning in bimanual suture needle regrasping [4], and Aquatic Navigation [5]. Each one of these applications employs RL as a motivating alternative to automating manual labor.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Parameter Optimization Using Genetic Algorithm in Deep Reinforcement Learning for Robotic Manipulation Tasks

Sehgal¹,

Ward²,

La³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning agents can make use of Reinforcement Learning (RL) to decide their actions by using a reward function. However, the learning process is greatly influenced by the elect of values of the parameters used in the learning algorithm. This work proposed a Deep Deterministic Policy Gradient (DDPG) and Hindsight Experience Replay (HER) based method, which makes use of the Genetic Algorithm (GA) to fine-tune the parameters' values. This method (GA-DRL) experimented on six robotic manipulation tasks: fetch-reach; fetchslide; fetch-push; fetch-pick and place; door-opening; and aubo-reach. Analysis of these results demonstrated a significant increase in performance and a decrease in learning time. Also, we compare and provide evidence that GA-DRL is better than the existing methods.

show abstract

Section: Ddpg + Her and Gamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Automatic Parameter Optimization Using Genetic Algorithm in Deep Reinforcement Learning for Robotic Manipulation Tasks

Sehgal¹,

Ward²,

La³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To date, existing techniques for improving the safety of robotic systems rely mostly on Lagrangian multipliers [39], [53], [57], and do not provide formal safety guarantees, but rather optimize the training in an attempt to learn the required policies [12]. Other, more formal approaches focus solely on the systems' input-output relations [13], [42], without considering multiple invocations of the agent and its interactions with the environment. Thus, previous methods are not able to provide rigorous guarantees regarding the correctness of multistep robotic systems, and do not take into account sequential decision making -which renders these methods insufficient for detecting various safety and liveness violations.…”

Section: Related Workmentioning

confidence: 99%

“…To perform the training of our robot, we rely on Unity3D, a the popular engine originally designed for game development, that has recently been adopted for robotics simulation [42], [50]. In particular, the built-in physics engine, the powerful 3D rendering algorithm and the time control system (which allows to speed up the simulation by more than 10 times), have made Unity3D a very powerful tool in these contexts [29].…”

Section: Appendix B Technical Specifications Of the Robotmentioning

confidence: 99%

Verifying Learning-Based Robotic Navigation Systems

Amir¹,

Corsi²,

Yerushalmi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) has become a dominant deep-learning paradigm for various tasks in which complex policies are learned within reactive systems. In parallel, there has recently been significant research on verifying deep neural networks. However, to date, there has been little work demonstrating the use of modern verification tools on real, DRLcontrolled systems. In this case-study paper, we attempt to begin bridging this gap, and focus on the important task of mapless robotic navigation -a classic robotics problem, in which a robot, usually controlled by a DRL agent, needs to efficiently and safely navigate through an unknown arena towards a desired target. We demonstrate how modern verification engines can be used for effective model selection, i.e., the process of selecting the best available policy for the robot in question from a pool of candidate policies. Specifically, we use verification to detect and rule out policies that may demonstrate suboptimal behavior, such as collisions and infinite loops. We also apply verification to identify models with overly conservative behavior, thus allowing users to choose superior policies that are better at finding an optimal, shorter path to a target. To validate our work, we conducted extensive experiments on an actual robot, and confirmed that the suboptimal policies detected by our method were indeed flawed. We also compared our verification-driven approach to state-of-the-art gradient attacks, and our results demonstrate that gradient-based methods are inadequate in this setting. Our work is the first to demonstrate the use of DNN verification backends for recognizing suboptimal DRL policies in real-world robots, and for filtering out unwanted policies. We believe that the methods presented in this work can be applied to a large range of application domains that incorporate deep-learning-based agents.

show abstract

“…Recently, Reinforcement Learning (RL) [1] has been put to significant uses such as robotic table tennis [2], surgical robot planning [3], rapid motion planning in bimanual regrasping for suture needles [4] and Aquatic Navigation [5]. Each of these applications make use of RL as an encouraging substitute to automating manual effort.…”

Section: Introductionmentioning

confidence: 99%

GA-DRL: Genetic Algorithm-Based Function Optimizer in Deep Reinforcement Learning for Robotic Manipulation Tasks

Sehgal¹,

Ward²,

La³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reinforcement learning (RL) enables agents to make a decision based on a reward function. However, in the process of learning, the choice of values for learning algorithm parameters can significantly impact the overall learning process. In this paper, we proposed a Genetic Algorithm-based Deep Deterministic Policy Gradient and Hindsight Experience Replay method (called GA-DRL) to find near-optimal values of learning parameters. We used the proposed GA-DRL method on fetch-reach, slide, push, pick and place, and door opening in robotic manipulation tasks. With some modifications, our proposed GA-DRL method was also applied to the auboreach environment. Our experimental evaluation shows that our method leads to significantly better performance, faster than the original algorithm. Also, we provide evidence that GA-DRL performs better than the existing methods.

show abstract

Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigation

Cited by 20 publications

References 16 publications

Automatic Parameter Optimization Using Genetic Algorithm in Deep Reinforcement Learning for Robotic Manipulation Tasks

Automatic Parameter Optimization Using Genetic Algorithm in Deep Reinforcement Learning for Robotic Manipulation Tasks

Verifying Learning-Based Robotic Navigation Systems

GA-DRL: Genetic Algorithm-Based Function Optimizer in Deep Reinforcement Learning for Robotic Manipulation Tasks

Contact Info

Product

Resources

About