The complexity of reachability in parametric Markov decision processes

Junges, Sebastian; Katoen, Joost-Pieter; Pérez, Guillermo A.; Winkler, Tobias

doi:10.1016/j.jcss.2021.02.006

Cited by 18 publications

(10 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Optimization-based approaches use branch-and-bound (Meuleau et al, 1999) or segmentation into reachable sets (Sharan & Burdick, 2014) to limit the searchable space. Junges et al (2018) construct an FSC using parameter synthesis for Markov chains, which is known to be ETR-complete (Junges, Katoen, et al, 2021), whereas NP ⊆ ETR ⊆ PSPACE. Carr et al (2018) render common POMDP scenarios as arcade games to capture human preferences that are formally cast into FSCs and subsequently verified.…”

Section: Related Workmentioning

confidence: 99%

Task-Aware Verifiable RNN-Based Policies for Partially Observable Markov Decision Processes

Carr

Jansen

Topcu

2021

jair

View full text Add to dashboard Cite

Partially observable Markov decision processes (POMDPs) are models for sequential decision-making under uncertainty and incomplete information. Machine learning methods typically train recurrent neural networks (RNN) as effective representations of POMDP policies that can efficiently process sequential data. However, it is hard to verify whether the POMDP driven by such RNN-based policies satisfies safety constraints, for instance, given by temporal logic specifications. We propose a novel method that combines techniques from machine learning with the field of formal methods: training an RNN-based policy and then automatically extracting a so-called finite-state controller (FSC) from the RNN. Such FSCs offer a convenient way to verify temporal logic constraints. Implemented on a POMDP, they induce a Markov chain, and probabilistic verification methods can efficiently check whether this induced Markov chain satisfies a temporal logic specification. Using such methods, if the Markov chain does not satisfy the specification, a byproduct of verification is diagnostic information about the states in the POMDP that are critical for the specification. The method exploits this diagnostic information to either adjust the complexity of the extracted FSC or improve the policy by performing focused retraining of the RNN. The method synthesizes policies that satisfy temporal logic specifications for POMDPs with up to millions of states, which are three orders of magnitude larger than comparable approaches.

show abstract

Section: Related Workmentioning

confidence: 99%

Task-Aware Verifiable RNN-Based Policies for Partially Observable Markov Decision Processes

Carr

Jansen

Topcu

2021

jair

View full text Add to dashboard Cite

show abstract

“…The reachability function can grow exponentially in the number of parameters, even for acyclic pMCs [1]. Feasibility is a computationally hard problem: finding parameter values for a pMC that satisfy a reachability objective is ETRcomplete [34] 4 . Feasibility has been tackled using sampling search methods such as PSO and Markov Chain Monte Carlo [16] and solving a non-linear optimization problem [4].…”

Section: Computing Reachability Functions Compute the Rational Functi...mentioning

confidence: 99%

“…Typical objectives are of the form: is the probability to reach some states below (or above) a given threshold q? The complexity of various pMC synthesis problems is studied in [1,34].…”

Section: Introductionmentioning

confidence: 99%

Fine-Tuning the Odds in Bayesian Networks

Salmani¹,

Katoen²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper proposes various new analysis techniques for Bayes networks in which conditional probability tables (CPTs) may contain symbolic variables. The key idea is to exploit scalable and powerful techniques for synthesis problems in parametric Markov chains. Our techniques are applicable to arbitrarily many, possibly dependent parameters that may occur in various CPTs. This lifts the severe restrictions on parameters, e.g., by restricting the number of parametrized CPTs to one or two, or by avoiding parameter dependencies between several CPTs, in existing works for parametric Bayes networks (pBNs). We describe how our techniques can be used for various pBN synthesis problems studied in the literature such as computing sensitivity functions (and values), simple and difference parameter tuning, ratio parameter tuning, and minimal change tuning. Experiments on several benchmarks show that our prototypical tool built on top of the probabilistic model checker Storm can handle several hundreds of parameters.

show abstract

“…The challenge in applying parameter synthesis is twofold: whereas the problem is ETR-complete 4 [34], the number of parameters grows linear in the number of different observations and the number of actions available to the controller. For many real-life applications we must thus deal with thousands of parameters.…”

Section: Introductionmentioning

confidence: 99%

“…Technically, we use graph-preserving to ensure continuously differentiability of ER s→ M . For acyclic pMCs, these functions are continuously differentiable without assuming graph-preservation[34].…”

mentioning

confidence: 99%

Gradient-Descent for Randomized Controllers under Partial Observability

Heck¹,

Spel²,

Junges³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Randomization is a powerful technique to create robust controllers, in particular in partially observable settings. The degrees of randomization have a significant impact on the system performance, yet they are intricate to get right. The use of synthesis algorithms for parametric Markov chains (pMCs) is a promising direction to support the design process of such controllers. This paper shows how to define and evaluate gradients of pMCs. Furthermore, it investigates varieties of gradient descent techniques from the machine learning community to synthesize the probabilities in a pMC. The resulting method scales to significantly larger pMCs than before and empirically outperforms the state-of-the-art, often by at least one order of magnitude.

show abstract

The complexity of reachability in parametric Markov decision processes

Cited by 18 publications

References 44 publications

Task-Aware Verifiable RNN-Based Policies for Partially Observable Markov Decision Processes

Task-Aware Verifiable RNN-Based Policies for Partially Observable Markov Decision Processes

Fine-Tuning the Odds in Bayesian Networks

Gradient-Descent for Randomized Controllers under Partial Observability

Contact Info

Product

Resources

About