DSMC Evaluation Stages: Fostering Robust and Safe Behavior in Deep Reinforcement Learning

Gros, Timo P.; Höller, Daniel; Hoffmann, Jörg; Klauck, Michaela; Meerkamp, Hendrik; Wolf, Verena

doi:10.1007/978-3-030-85172-9_11

Cited by 8 publications

(14 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from that, we plan to build upon MoGym to develop DSMC techniques further. With DSMC Evaluation Stages [21] it has already been shown that DSMC can be applied during deep RL to determine state space regions with weak performance to concentrate on them during the learning process. With the help of MoGym this technique can now be done much more integrated and there is room for further implementations into this direction in our tool chain.…”

Section: Discussionmentioning

confidence: 99%

“…As such, the environment provides a stable and fully controllable training and checking context to assert the safety risk induced by an agent during and after training. More concrete, MoGym leverages deep statistical model checking (DSMC) [20,21]. As shown in these works on DSMC, the quality assessment of an agent during training is not trivial and can especially not always be derived from the observed training returns.…”

Section: Introductionmentioning

confidence: 99%

“…As shown in these works on DSMC, the quality assessment of an agent during training is not trivial and can especially not always be derived from the observed training returns. Hence, analyzing the quality of the decision-making agents after training clearly is of interest [20,21], especially for badly interpretable agent structures such as neural networks (NN). In DSMC this is done by using the decision-making agent as an oracle resolving the non-determinism in the MDP specifying the environment.…”

Section: Introductionmentioning

confidence: 99%

“…-The DSMC API, also newly implemented on top of Momba. It includes a Python API to use the DSMC functionality [20,21] of the Modest Toolset [13,30]. -DSMC implemented in the Modest Toolset.…”

Section: Introductionmentioning

confidence: 99%

“…-DSMC implemented in the Modest Toolset. In prior work [20,21] We are not aware of any other work that enables a direct connection of formal verification models and reinforcement learning that directly allows the analysis of different RL agents for a variety of verification benchmarks.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

MoGym: Using Formal Models for Training and Verifying Decision-making Agents

Gros

Hermanns

Hoffmann

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

MoGym, is an integrated toolbox enabling the training and verification of machine-learned decision-making agents based on formal models, for the purpose of sound use in the real world. Given a formal representation of a decision-making problem in the JANI format and a reach-avoid objective, MoGym (a) enables training a decision-making agent with respect to that objective directly on the model using reinforcement learning (RL) techniques, and (b) it supports rigorous assessment of the quality of the induced decision-making agent by means of deep statistical model checking (DSMC). MoGym implements the standard interface for training environments established by OpenAI Gym, thereby connecting to the vast body of existing work in the RL community. In return, it makes accessible the large set of existing JANI model checking benchmarks to machine learning research. It thereby contributes an efficient feedback mechanism for improving in particular reinforcement learning algorithms. The connective part is implemented on top of Momba. For the DSMC quality assurance of the learned decision-making agents, a variant of the statistical model checker modes of the Modest Toolset is leveraged, which has been extended by two new resolution strategies for non-determinism when encountered during statistical evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MoGym: Using Formal Models for Training and Verifying Decision-making Agents

Gros

Hermanns

Hoffmann

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

The Modest State of Learning, Sampling, and Verifying Strategies

Hartmanns

Klauck

2022

Leveraging Applications of Formal Methods, Verification and Validation. Adaptation and Learning

Self Cite

View full text Add to dashboard Cite

Optimal decision-making under stochastic uncertainty is a core problem tackled in artificial intelligence/machine learning (AI), planning, and verification. Planning and AI methods aim to find good or optimal strategies to maximise rewards or the probability of reaching a goal. Verification approaches focus on calculating the probability or reward, obtaining the strategy as a side effect. In this paper, we connect three strands of work on obtaining strategies implemented in the context of the Modest Toolset: statistical model checking with either lightweight scheduler sampling or deep learning, and probabilistic model checking. We compare their different goals and abilities, and show newly extended experiments on Racetrack benchmarks that highlight the tradeoffs between the methods. We conclude with an outlook on improving the existing approaches and on generalisations to continuous models, and emphasise the need for further tool development to integrate methods that find, evaluate, compare, and explain strategies.

show abstract