Widest Paths and Global Propagation in Bounded Value Iteration for Stochastic Games

Phalakarn, Kittiphon; Takisaka, Toru; Haas, Thomas; Hasuo, Ichiro

doi:10.1007/978-3-030-53291-8_19

Cited by 9 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Numerical algorithms, in particular ones that converge in the limit, are prone to delivering wrong results. For VI, the recognition of this problem has led to a series of improvements over the last decade [8,34,40,19,54,56]. We show that PI faces a similar problem.…”

Section: Introductionmentioning

confidence: 79%

See 1 more Smart Citation

A Practitioner’s Guide to MDP Model Checking Algorithms

Hartmanns

Junges

Quatmann

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Model checking undiscounted reachability and expected-reward properties on Markov decision processes (MDPs) is key for the verification of systems that act under uncertainty. Popular algorithms are policy iteration and variants of value iteration; in tool competitions, most participants rely on the latter. These algorithms generally need worst-case exponential time. However, the problem can equally be formulated as a linear program, solvable in polynomial time. In this paper, we give a detailed overview of today’s state-of-the-art algorithms for MDP model checking with a focus on performance and correctness. We highlight their fundamental differences, and describe various optimizations and implementation variants. We experimentally compare floating-point and exact-arithmetic implementations of all algorithms on three benchmark sets using two probabilistic model checkers. Our results show that (optimistic) value iteration is a sensible default, but other algorithms are preferable in specific settings. This paper thereby provides a guide for MDP verification practitioners—tool builders and users alike.

show abstract

Section: Introductionmentioning

confidence: 79%

“…Instead, for years, implementations used a naive stopping criterion that could return arbitrarily wrong results [33]. This problem's discovery sparked the development of sound variants of VI [8,34,40,19,54,56], including interval iteration, sound value iteration, and optimistic value iteration. A sound VI algorithm guarantees…”

Section: Guaranteesmentioning

confidence: 99%

A Practitioner’s Guide to MDP Model Checking Algorithms

Hartmanns

Junges

Quatmann

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Numerical algorithms, in particular ones that converge in the limit, are prone to delivering wrong results. For VI, the recognition of this problem has led to a series of improvements over the last decade [8,26,29,33,41,43]. We show that PI faces a similar problem.…”

Section: Introductionmentioning

confidence: 79%

“…Instead, for years, implementations 0 Mn: used a naive stopping criterion that could return arbitrarily wrong results [25]. This problem's discovery [25] sparked the development of various sound variants of VI [8,26,29,33,41,43], including interval iteration, sound value iteration, and optimistic value iteration. A sound VI algorithm guarantees ε-precise results, i.e.…”

Section: Guaranteesmentioning

confidence: 99%

A Practitioner's Guide to MDP Model Checking Algorithms

Hartmanns¹,

Junges²,

Quatmann³

et al. 2023

Preprint

View full text Add to dashboard Cite

Model checking undiscounted reachability and expected-reward properties on Markov decision processes (MDPs) is key for the verification of systems that act under uncertainty. Popular algorithms are policy iteration and variants of value iteration; in tool competitions, most participants rely on the latter. These algorithms generally need worst-case exponential time. However the problem can equally be formulated as a linear program, solvable in polynomial time. In this paper, we give a detailed overview of today's state-of-the-art algorithms for MDP model checking with a focus on performance and correctness. We highlight their fundamental differences, and describe various optimisations and implementation variants. We experimentally compare floating-point and exact-arithmetic implementations of all algorithms on three benchmark sets using two probabilistic model checkers. Our results show that (optimistic) value iteration is a sensible default, but other algorithms are preferable in specific settings. This paper thereby provides a guide for MDP verification practitioners-tool builders and users alike.

show abstract

Widest Paths and Global Propagation in Bounded Value Iteration for Stochastic Games

Phalakarn

Takisaka

Haas

et al. 2020

Computer Aided Verification

Self Cite

View full text Add to dashboard Cite

Solving stochastic games with the reachability objective is a fundamental problem, especially in quantitative verification and synthesis. For this purpose, bounded value iteration (BVI) attracts attention as an efficient iterative method. However, BVI's performance is often impeded by costly end component (EC) computation that is needed to ensure convergence. Our contribution is a novel BVI algorithm that conducts, in addition to local propagation by the Bellman update that is typical of BVI, global propagation of upper bounds that is not hindered by ECs. To conduct global propagation in a computationally tractable manner, we construct a weighted graph and solve the widest path problem in it. Our experiments show the algorithm's performance advantage over the previous BVI algorithms that rely on EC computation.

show abstract

Widest Paths and Global Propagation in Bounded Value Iteration for Stochastic Games

Cited by 9 publications

References 24 publications

A Practitioner’s Guide to MDP Model Checking Algorithms

A Practitioner’s Guide to MDP Model Checking Algorithms

A Practitioner's Guide to MDP Model Checking Algorithms

Widest Paths and Global Propagation in Bounded Value Iteration for Stochastic Games

Contact Info

Product

Resources

About