Trainify: A CEGAR-Driven Training and Verification Framework for Safe Deep Reinforcement Learning

Abstract

Deep Reinforcement Learning (DRL) has demonstrated its strength in developing intelligent systems. These systems shall be formally guaranteed to be trustworthy when applied to safety-critical domains, which is typically achieved by formal verification performed after training. This train-then-verify process has two limits: (i) trained systems are difficult to formally verify due to their continuous and infinite state space and inexplicable AI components (i.e., deep neural networks), and (ii) the ex post facto detection of bugs increases both the time- and money-wise cost of training and deployment. In this paper, we propose a novel verification-in-the-loop training framework called Trainify for developing safe DRL systems driven by counterexample-guided abstraction and refinement. Specifically, Trainify trains a DRL system on a finite set of coarsely abstracted but efficiently verifiable state spaces. When verification fails, we refine the abstraction based on returned counterexamples and train again on the finer abstract states. The process is iterated until all predefined properties are verified against the trained system. We demonstrate the effectiveness of our framework on six classic control systems. The experimental results show that our framework yields more reliable DRL systems with provable guarantees without sacrificing system performance such as cumulative reward and robustness than conventional DRL approaches.

Model Checking for Adversarial Multi-Agent Reinforcement Learning with Reactive Defense Methods

“…Optimizing the incremental model building process of COOL-MC can increase the number of supported CMARL agents. Incorporating safe CMARL approaches would also be valuable extensions to our method, as already done in the single RL domain (Carr et al 2023;Jin et al 2022;Jothimurugan et al 2022;Jansen et al 2020).…”

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

“…We presented the tool COOL-MC, which provides a tight interaction between model checking and reinforcement learning. In the future, we will extend the tool to directly incorporate safe reinforcement learning approaches [22,[25][26][27] and will extend the model expressivity to partially observable MDPs [29].…”

Robustness Verification of Deep Reinforcement Learning Based Control Systems Using Reward Martingales

“…An Analytic Approach We propose an analytical approach for the control policies that are trained on discretized abstract states. In previous work (Jin et al 2022;Tian et al 2022;Li et al 2022), a compact but infinitely continuous state space S was discretized to a finite set of abstract states, i.e., S = L i=1 S i and ∀i j, S i ∩S j = ∅. Then a neural network policy π was trained on the set of abstract states.…”

“…If policies are implemented by DNNs on infinite and continuous state space, we take advantage of the overapproximation-based method (Lechner et al 2022). We also propose an analytical method to compute expected values precisely when policies are trained on discretized abstract state space in recently emerging approaches (Jin et al 2022;Li et al 2022;Drews, Albarghouthi, and D'Antoni 2020).…”

“…Qualitative Verification of DRL-based Control SystemsFormal verification of DRL-based control systems has received increasing attention for safety assurance in recent years Jin et al (2022). proposed a CEGAR-driven training and verification framework that guarantees that the trained systems satisfy the properties predefined in ACTL formulas Bacci (2022).…”