Approximate Abstractions of Markov Chains with Interval Decision Processes

Lun, Yuriy Zacchia; Wheatley, Jack; D’Innocenzo, Alessandro; Abate, Alessandro

doi:10.1016/j.ifacol.2018.08.016

Cited by 11 publications

(13 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Approximate abstractions of dt-SCS with interval MDPs are proposed in [ZLWDA18]. The abstraction leverages the semantics of IMDPs and the standard notion of approximate probabilistic bisimulation.…”

Section: Abstractions Formentioning

confidence: 99%

Automated Verification and Synthesis of Stochastic Hybrid Systems: A Survey

Lavaei¹,

Soudjani²,

Abate³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Stochastic hybrid systems have received significant attentions as a relevant modelling framework describing many systems, from engineering to the life sciences: they enable the study of numerous applications, including transportation networks, biological systems and chemical reaction networks, smart energy and power grids, and beyond. Automated verification and policy synthesis for stochastic hybrid systems can be inherently challenging: this is due to the heterogeneity of their dynamics (presence of continuous and discrete components), the presence of uncertainty, and in some applications the large dimension of state and input sets. Over the past few years, a few hundred articles have investigated these models, and developed diverse and powerful approaches to mitigate difficulties encountered in the analysis and synthesis of such complex stochastic systems. In this survey, we overview the most recent results in the literature and discuss different approaches, including (in)finite abstractions, verification and synthesis for temporal logic specifications, stochastic similarity relations, (control) barrier certificates, compositional techniques, and a selection of results on continuous-time stochastic systems; we finally survey recently developed software tools that implement the discussed approaches. Throughout the manuscript we discuss a few open topics to be considered as potential future research directions: we hope that this survey will guide younger researchers through a comprehensive understanding of the various challenges, tools, and solutions in this enticing and rich scientific area.

show abstract

Section: Abstractions Formentioning

confidence: 99%

Automated Verification and Synthesis of Stochastic Hybrid Systems: A Survey

Lavaei¹,

Soudjani²,

Abate³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…There is also a body of work on verifying continuous space probabilistic models and stochastic hybrid systems, by building finite-state abstractions as, e.g., interval Markov chains [32] or interval MDPs [35,10], but these do not consider control policies encoded as neural networks. Similarly, abstractions of discrete-state probabilistic models use similar ideas to our approach, notably via the use of interval Markov chains [14] and stochastic games [26].…”

Section: Related Workmentioning

confidence: 99%

Probabilistic Guarantees for Safe Deep Reinforcement Learning

Bacci

Parker

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Deep reinforcement learning is an increasingly popular technique for synthesising policies to control an agent's interaction with its environment. There is also growing interest in formally verifying that such policies are correct and execute safely. Progress has been made in this area by building on existing work for verification of deep neural networks and of continuous-state dynamical systems. In this paper, we tackle the problem of verifying probabilistic policies for deep reinforcement learning, which are used to, for example, tackle adversarial environments, break symmetries and manage trade-offs. We propose an abstraction approach, based on interval Markov decision processes, that yields probabilistic guarantees on a policy's execution, and present techniques to build and solve these models using abstract interpretation, mixed-integer linear programming, entropy-based refinement and probabilistic model checking. We implement our approach and illustrate its effectiveness on a selection of reinforcement learning benchmarks.

show abstract

“…A Euclidean MDP M = (S, G, Act, T , C) and a partition A of S induces an abstracting IMDP [10,11] according to the following definition.…”

Section: Approximation By Partitioningmentioning

confidence: 99%

“…Upper and lower bounds for reachability probabilities are obtained from the abstract MDP, which is formalized as a two player stochastic game. [11] is concerned with obtaining accurate specifications of an abstraction obtained by state space partitioning. The underlying state space is finite, and a fixed partition is given.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Approximating Euclidean by Imprecise Markov Decision Processes

Jaeger¹,

Bacci²,

Larsen³

et al. 2020

Preprint

View full text Add to dashboard Cite

Euclidean Markov decision processes are a powerful tool for modeling control problems under uncertainty over continuous domains. Finite state imprecise, Markov decision processes can be used to approximate the behavior of these infinite models. In this paper we address two questions: first, we investigate what kind of approximation guarantees are obtained when the Euclidean process is approximated by finite state approximations induced by increasingly fine partitions of the continuous state space. We show that for cost functions over finite time horizons the approximations become arbitrarily precise. Second, we use imprecise Markov decision process approximations as a tool to analyse and validate cost functions and strategies obtained by reinforcement learning. We find that, on the one hand, our new theoretical results validate basic design choices of a previously proposed reinforcement learning approach. On the other hand, the imprecise Markov decision process approximations reveal some inaccuracies in the learned cost functions.

show abstract

Approximate Abstractions of Markov Chains with Interval Decision Processes

Cited by 11 publications

References 9 publications

Automated Verification and Synthesis of Stochastic Hybrid Systems: A Survey

Automated Verification and Synthesis of Stochastic Hybrid Systems: A Survey

Probabilistic Guarantees for Safe Deep Reinforcement Learning

Approximating Euclidean by Imprecise Markov Decision Processes

Contact Info

Product

Resources

About