Verifiably Safe Off-Model Reinforcement Learning

Abstract

The desire to use reinforcement learning in safety-critical settings has inspired a recent interest in formal methods for learning algorithms. Existing formal methods for learning and optimization primarily consider the problem of constrained learning or constrained optimization. Given a single correct model and associated safety constraint, these approaches guarantee efficient learning while provably avoiding behaviors outside the safety constraint. Acting well given an accurate environmental model is an important prerequisite for safe learning, but is ultimately insufficient for systems that operate in complex heterogeneous environments. This paper introduces verification-preserving model updates, the first approach toward obtaining formal safety guarantees for reinforcement learning in settings where multiple possible environmental models must be taken into account. Through a combination of inductive data and deductive proving with design-time model updates and runtime model falsification, we provide a first approach toward obtaining formal safety proofs for autonomous systems acting in heterogeneous environments. This research was sponsored by the Defense Advanced Research Projects Agency (DARPA) under grant number FA8750-18-C-0092. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government or any other entity.

NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems

“…In other learning-enabled systems, falsification and testing-based approaches [12,13,45] have shown a significant promise in enhancing the safety of systems where perception components and neural networks interact with the physical world. Finally, there is significant related work in the domain of safe reinforcement learning [2,15,47,59], and combining guarantees from NNV with those provided in these methods would be interesting to explore.…”

Logical Foundations of Cyber-Physical Systems

“…In that case do quantitative versions of ModelPlex monitors serve as reward signals that tend to pull the system back from unsafe into safe space. Beyond the experimental observation that this enables a safe recovery outside well-modeled parts of the world is it possible to give rigorous safety proofs of the resulting behavior of the learning CPS for the case of multiple possible models that are not all wrong or that can be modified to safely fit reality with verification-preserving model updates [Fulton and Platzer, 2019]. The basic idea is to use the conjunction of all ModelPlex monitors of plausible models to determine which action is safe while discarding models whose predictions did not end up happening.…”

“…KeYmaera X implements the transfer of safety proofs for CPS models to CPS implementations by synthesizing provably correct runtime monitors with Model-Plex [Mitsch and Platzer, 2016b], which result in CPS executables that are formally verified in a chain of theorem provers [Bohrer et al, 2018]. ModelPlex is also the basis for enabling safe artificial intelligence in cyber-physical systems [Platzer, 2019a] by wrapping reinforcement learning in a verified safety sandbox [Fulton and Platzer, 2018] and steering back toward safety when outside the confounds of a well-modeled part of the system behavior [Fulton and Platzer, 2019]. These results that will be surveyed here provide rigorous safety guarantees for CPS implementations (even those that involve machine learning) without having to deal with the entire complexity of their implementation during verification.…”

Verifiably safe SCUBA diving using commodity sensors

“…The diver's oxygen consumption will depend upon both heart rate and several user-specific parameters (b, hr ss , and τ ) fit using data from experiments [4,7]. Future work will incorporate system identification into this model using [2] and will evaluate a prototype based on our model.…”