Improving modified policy iteration for probabilistic model checking

Abstract

Value iteration, policy iteration and their modified versions are well-known algorithms for probabilistic model checking of Markov Decision Processes. One the challenge of these methods is that they are time-consuming in most cases. Several techniques have been proposed to improve the performance of iterative methods for probabilistic model checking. However, the running time of these techniques depends on the graphical structure of the model and in some cases their performance is worse than the performance of the standard methods. In this paper, we propose two new heuristics to accelerate the modified policy iteration method. We first define a criterion for the usefulness of the computations of each iteration of this method. The first contribution of our work is to develop and use a criterion to reduce the number of iterations in modified policy iteration. As the second contribution, we propose a new approach to identify useless updates in each iteration. This method reduces the running time of computations by avoiding useless updates of states. The proposed heuristics have been implemented in the PRISM model checker and applied on several standard case studies. We compare the running time of our heuristics with the running time of previous standard and improved methods. Experimental results show that our techniques yields a significant speed-up.

State ordering and classification for analyzing non-sparse large Markov Models

“…The modified policy iteration [1] method updates the policies after a fixed number of iterations (100 iterations, for example) rather than satisfying the convergence criterion. It usually causes faster convergence to the estimations of values [29,32]. More details of modified policy iteration are available in [1,19,31].…”

“…Although modified policy iteration is usually faster than the standard policy iteration (Algorithm 1), the number of iterations of the Gauss-Seidel method influences its running time. In [29] a dynamic method has been proposed for determining the number of iterations for every quotient DTMCs.…”

“…The RTDP approach and its extensions have also been used in probabilistic model checking to improve the performance of the iterative computations [25] or guarantee the precision of the computed values [15,26]. Graph-based prioritizing methods have been proposed in [27][28][29][30] to provide more appropriate state ordering and avoid useless updates. The main goal of these approaches is to accelerate the convergence to the optimal policy that an intelligent agent should utilize to maximize its performance, or reduce the running time of computing the optimal probability of reaching some special state from the initial one.…”

“…The main goal of these approaches is to accelerate the convergence to the optimal policy that an intelligent agent should utilize to maximize its performance, or reduce the running time of computing the optimal probability of reaching some special state from the initial one. Focusing on the policy iteration method as one of the standard iterative approaches for analyzing MDP models, some improved methods have been proposed in [29,31] to avoid useless updates and provide faster convergence to the optimal values. Although in most cases, the proposed prioritizing methods reduce the running time of computations by one or two orders of magnitude, their success depends on the graphical structure of the models and their performance degrades for non-sparse models [30].…”