COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

Gross, Dennis C.; Jansen, Nils; Junges, Sebastian; Pérez, Guillermo A.

doi:10.1007/978-3-031-21213-0_3

Cited by 5 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our method, on the other hand, gives us a reachability probability done = 0.58 (see Table 1). However, at some point, our model checking method is also limited by the size of the induced DTMC and runs out of memory (Gross et al 2022).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Model Checking for Adversarial Multi-Agent Reinforcement Learning with Reactive Defense Methods

Gross¹,

Schmidl²,

Jansen³

et al. 2023

ICAPS

View full text Add to dashboard Cite

Cooperative multi-agent reinforcement learning (CMARL) enables agents to achieve a common objective. However, the safety (a.k.a. robustness) of the CMARL agents operating in critical environments is not guaranteed. In particular, agents are susceptible to adversarial noise in their observations that can mislead their decision-making. So-called denoisers aim to remove adversarial noise from observations, yet, they are often error-prone. A key challenge for any rigorous safety verification technique in CMARL settings is the large number of states and transitions, which generally prohibits the construction of a (monolithic) model of the whole system. In this paper, we present a verification method for CMARL agents in settings with or without adversarial attacks or denoisers. Our method relies on a tight integration of CMARL and a verification technique referred to as model checking. We showcase the applicability of our method on various benchmarks from different domains. Our experiments show that our method is indeed suited to verify CMARL agents and that it scales better than a naive approach to model checking.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Recall, the joint policy π induced by the set of all agent policies {π i } i∈I is a single policy π (Boutilier 1996). The tool COOL-MC 1 (Gross et al 2022) allows model checking of a single RL policy against a user-provided PCTL property and MDP. Thereby, it builds the induced DTMC incrementally (Cassez et al 2005).…”

Section: Model Checking Of Cmarl Agentsmentioning

confidence: 99%

Model Checking for Adversarial Multi-Agent Reinforcement Learning with Reactive Defense Methods

Gross¹,

Schmidl²,

Jansen³

et al. 2023

ICAPS

View full text Add to dashboard Cite

show abstract

“…Summary. These results are part of the publications in [15,29,35,38,55]. A common approach to safe reinforcement learning is to employ a so-called shield that forces an RL agent to select only safe actions.…”

Section: Safe Deep Reinforcement Learningmentioning

confidence: 99%

Intelligent and Dependable Decision-Making Under Uncertainty

Jansen

2023

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This talk highlights our vision of foundational and application-driven research toward safety, dependability, and correctness in artificial intelligence (AI). We take a broad stance on AI that combines formal methods, machine learning, and control theory. As part of this research line, we study problems inspired by autonomous systems, planning in robotics, and industrial applications. We consider reinforcement learning (RL) as a specific machine learning technique for decisionmaking under uncertainty. RL generally learns to behave optimally via trial and error. Consequently, and despite its massive success in the past years, RL lacks mechanisms to ensure safe and correct behavior. Formal methods, in particular formal verification, is a research area that provides formal guarantees of a system's correctness and safety based on rigorous methods and precise specifications. Yet, fundamental challenges have obstructed the effective application of verification to reinforcement learning. Our main objective is to devise novel, data-driven verification methods that tightly integrate with RL. In particular, we develop techniques that address real-world challenges to the safety of AI systems in general: Scalability, expressiveness, and robustness against the uncertainty that occurs when operating in the real world. The overall goal is to advance the real-world deployment of reinforcement learning. Synopsis: Robust and Dependable Artificial IntelligenceArtificial intelligence (AI) is a disruptive force. Most major technology companies employ or develop AI, and with growing applications in fields like healthcare [37], transportation [48,68], game playing [51], finance [9], or robotics in general [44], it is entering our everyday lives. We can expect that our societal and technological involvement with AI will only intensify in the future. Such tight interaction with AI requires serious safety and correctness considerations. Recently, the field of safety in AI has triggered a vast amount of research with several seminal works defining their view on this area [4,25,58,61].Can Formal Verification Help to Ensure AI Safety? The area of formal methods offers structured and rigorous ways to reason about the correctness N.

show abstract

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

Gross

Jansen

Junges

et al. 2022

Dependable Software Engineering. Theories, Tools, and Applications

Self Cite

View full text Add to dashboard Cite

This paper presents COOL-MC, a tool that integrates stateof-the-art reinforcement learning (RL) and model checking. Specifically, the tool builds upon the OpenAI gym and the probabilistic model checker Storm. COOL-MC provides the following features: (1) a simulator to train RL policies in the OpenAI gym for Markov decision processes (MDPs) that are defined as input for Storm, (2) a new model builder for Storm, which uses callback functions to verify (neural network) RL policies, (3) formal abstractions that relate models and policies specified in OpenAI gym or Storm, and (4) algorithms to obtain bounds on the performance of so-called permissive policies. We describe the components and architecture of COOL-MC and demonstrate its features on multiple benchmark environments.

show abstract

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

Cited by 5 publications

References 34 publications

Model Checking for Adversarial Multi-Agent Reinforcement Learning with Reactive Defense Methods

Model Checking for Adversarial Multi-Agent Reinforcement Learning with Reactive Defense Methods

Intelligent and Dependable Decision-Making Under Uncertainty

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

Contact Info

Product

Resources

About