Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning

Abstract

The last half-decade has seen a steep rise in the number of contributions on safe learning methods for real-world robotic deployments from both the control and reinforcement learning communities. This article provides a concise but holistic review of the recent advances made in using machine learning to achieve safe decision making under uncertainties, with a focus on unifying the language and frameworks used in control theory and reinforcement learning research. Our review includes: learning-based control approaches that safely improve performance by learning the uncertain dynamics, reinforcement learning approaches that encourage safety or robustness, and methods that can formally certify the safety of a learned control policy. As dataand learning-based robot control methods continue to gain traction, researchers must understand when and how to best leverage them in real-world scenarios where safety is imperative, such as when operating in close proximity to humans. We highlight some of the open challenges that will drive the field of robot learning in the coming years, and emphasize the need for realistic physics-based benchmarks to facilitate fair comparisons between control and reinforcement learning approaches.

Learning Provably Robust Motion Planners Using Funnel Libraries

“…Embedding such structure would facilitate faster training and potentially better generalization. Another exciting future direction is to augment our approach with methods from safe reinforcement learning [56] to provide generalization guarantees on the success of safety-critical systems while ensuring safety during the training process itself (in contrast to only providing guarantees for deployment, as we do here). Finally, we are working towards hardware experiments of our approach on a drone navigating an obstacle field under wind disturbances.…”

Probabilistic design of optimal sequential decision-making algorithms in learning and control

“…Safety. We refer to [81] for a detailed review of learning-based control. Essentially, safety constraints can be embedded in Problem 2 in three ways, which in turn correspond to different safety levels [81].…”

Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning

“…Safe RL Our work is broadly related to the safe reinforcement learning and control literature; we refer interested readers to (Garcıa and Fernández 2015;Brunke et al 2021) for surveys on this topic. A popular class of approaches incorporates Lagrangian constraint regularization into the policy updates in policy-gradient algorithms (Achiam et al 2017;Ray, Achiam, and Amodei 2019;Tessler, Mankowitz, and Mannor 2018;Dalal et al 2018;Cheng et al 2019;Zhang, Vuong, and Ross 2020;Chow et al 2019).…”