The Steady-State Control Problem for Markov Decision Processes

Abstract

Abstract. This paper addresses a control problem for probabilistic models in the setting of Markov decision processes (MDP). We are interested in the steady-state control problem which asks, given an ergodic MDP M and a distribution δ goal , whether there exists a (history-dependent randomized) policy π π π ensuring that the steady-state distribution of M under π π π is exactly δ goal . We first show that stationary randomized policies suffice to achieve a given steady-state distribution. Then we infer that the steady-state control problem is decidable for MDP, and can be represented as a linear program which is solvable in PTIME. This decidability result extends to labeled MDP (LMDP) where the objective is a steady-state distribution on labels carried by the states, and we provide a PSPACE algorithm. We also show that a related steady-state language inclusion problem is decidable in EXPTIME for LMDP. Finally, we prove that if we consider MDP under partial observation (POMDP), the steady-state control problem becomes undecidable.

MultiGain: A Controller Synthesis Tool for MDPs with Multiple Mean-Payoff Objectives

“…In adversarial environments the problem reduces to games and for probabilistic environments to MDPs, with multiple mean-payoff objectives [16]. (B) The problem of synthesis of steady state distributions for ergodic MDPs was considered in [4]. The problem can model para.…”

“…be modeled with multiple mean-payoff objectives by considering indicator reward functions r s , for each state s, that assign reward 1 to every action enabled in s and 0 to all other actions. The steady state distribution synthesis question of [4] then reduces to the existence question for multiple mean-payoff MDPs.…”

LTL-Constrained Steady-State Policy Synthesis

“…The steady-state control was introduced in [Akshay et al, 2013], treating the case of recurrent MDP and showing the problem is in PSPACE by quadratic programming. It is combined with LRA reward maximization, giving rise to steadystate policy synthesis, in [Velasquez, 2019].…”

“…In terms of the automata representation, the policy is 2-memory, remembering whether a step has been already taken, see Fig. 5 in Appendix A. Consequently, memory may be necessary, in contrast to the claim of [Velasquez, 2019] that memoryless policies are sufficient by [Akshay et al, 2013], which holds only for the setting with recurrent chains. Moreover, the combination with LTL may require even unbounded memory: Example 2.…”

“…On the other hand, Steady-State Control (SSC, a.k.a. Steady-State Policy Synthesis) [Akshay et al, 2013] constrains the frequency with which states are visited, providing a more quantitative and more behavioural perspective (in terms of states of the system, as opposed to logic-based or reward-based specifications). Recently, it has started receiving more attention also in AI planning [Velasquez, 2019;Atia et al, 2020], improving the theoretical complexity and its applicability to a wider class of MDP (although still being quite restrictive on the class of policies, see below).…”