Shield Synthesis for Reinforcement Learning

Könighofer, Bettina; Lorber, Florian; Jansen, Nils; Bloem, Roderick

doi:10.1007/978-3-030-61362-4_16

Cited by 38 publications

(17 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several extensions exist [4,6,29,39]. The shielding approach has been shown to be successful in combination with RL [2,21]. Jansen et al [19] introduced offline shielding with respect to probabilistic safety.…”

Section: Related Workmentioning

confidence: 99%

Online Shielding for Stochastic Systems

Könighofer

Rudolf

Palmisano

et al. 2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

We propose a method to develop trustworthy reinforcement learning systems. To ensure safety especially during exploration, we automatically synthesize a correct-by-construction runtime enforcer, called a shield, that blocks all actions of the agent that are unsafe with respect to a temporal logic specification. Our main contribution is a new synthesis algorithm for computing the shield online. Existing offline shielding approaches compute exhaustively the safety of all states-action combinations ahead-of-time, resulting in huge computation times, large memory consumption, and significant delays at runtime due to the look-ups in huge databases. The intuition behind online shielding is to compute at runtime the set of all states that could be reached in the near future. For each of these states, the safety of all available actions is analysed and used for shielding as soon as one of the considered states is reached. Our proposed method is general and can be applied to a wide range of planning problems with stochastic behaviour. For our evaluation, we selected a 2player version of the classical computer game Snake. The game requires fast decisions and the multiplayer setting induces a large state space, computationally expensive to analyze exhaustively. The safety objective of collision avoidance is easily transferable to a variety of planning tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Online Shielding for Stochastic Systems

Könighofer

Rudolf

Palmisano

et al. 2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…These approaches leverage SMT solvers (e.g., [19]), based on LP and MILP solvers (e.g., [6], [10], [35], [42]), the propagation of symbolic intervals and abstract interpretation (e.g., [13], [43]- [45]), abstraction-refinement techniques (e.g., [3], [11]), and many others. Recent work has extended beyond answering yes/no questions about DNNs, targeting tasks such as automated DNN repair [15], [30] and quantitative verification [4]. Verification approaches have also been proposed for recurrent networks [23], [47], which could potentially also be simplified.…”

Section: Related Workmentioning

confidence: 99%

Pruning and Slicing Neural Networks using Formal Verification

Lahav,

Katz

2021

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) play an increasingly important role in various computer systems. In order to create these networks, engineers typically specify a desired topology, and then use an automated training algorithm to select the network's weights. While training algorithms have been studied extensively and are well understood, the selection of topology remains a form of art, and can often result in networks that are unnecessarily large -and consequently are incompatible with end devices that have limited memory, battery or computational power. Here, we propose to address this challenge by harnessing recent advances in DNN verification. We present a framework and a methodology for discovering redundancies in DNNs -i.e., for finding neurons that are not needed, and can be removed in order to reduce the size of the DNN. By using sound verification techniques, we can formally guarantee that our simplified network is equivalent to the original, either completely, or up to a prescribed tolerance. Further, we show how to combine our technique with slicing, which results in a family of very small DNNs, which are together equivalent to the original. Our approach can produce DNNs that are significantly smaller than the original, rendering them suitable for deployment on additional kinds of systems, and even more amenable to subsequent formal verification. We provide a proof-of-concept implementation of our approach, and use it to evaluate our techniques on several real-world DNNs.

show abstract

“…In addition to general DNN verification engines, methods have been devised to formally verify safety properties of DRL systems, which are the subject matter of this work. Such approaches include shield synthesis [31], and combining the verification process with verified runtime monitoring [16]. Other methods focus on finding adversarial attacks that pertain specifically to DRL agents, e.g., by using MILP [11].…”

Section: Related Workmentioning

confidence: 99%

Towards Scalable Verification of Deep Reinforcement Learning

Amir,

Schapira,

Katz

2021

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) have gained significant popularity in recent years, becoming the state of the art in a variety of domains. In particular, deep reinforcement learning (DRL) has recently been employed to train DNNs that act as control policies for various types of real-world systems. In this work, we present the whiRL 2.0 tool, which implements a new approach for verifying complex properties of interest for such DRL systems. To demonstrate the benefits of whiRL 2.0, we apply it to case studies from the communication networks domain that have recently been used to motivate formal verification of DRL systems, and which exhibit characteristics that are conducive for scalable verification. We propose techniques for performing k-induction and automated invariant inference on such systems, and use these techniques for proving safety and liveness properties of interest that were previously impossible to verify due to the scalability barriers of prior approaches. Furthermore, we show how our proposed techniques provide insights into the inner workings and the generalizability of DRL systems. whiRL 2.0 is publicly available online.

show abstract

Shield Synthesis for Reinforcement Learning

Cited by 38 publications

References 14 publications

Online Shielding for Stochastic Systems

Online Shielding for Stochastic Systems

Pruning and Slicing Neural Networks using Formal Verification

Towards Scalable Verification of Deep Reinforcement Learning

Contact Info

Product

Resources

About