One-Counter Markov Decision Processes

Abstract

We study the computational complexity of central analysis problems for One-Counter Markov Decision Processes (OC-MDPs), a class of finitely-presented, countable-state MDPs. OC-MDPs extend finite-state MDPs with an unbounded counter. The counter can be incremented, decremented, or not changed during each state transition, and transitions may be enabled or not depending on both the current state and on whether the counter value is 0 or not. Some states are "random", from where the next transition is chosen according to a given probability distribution, while other states are "controlled", from where the next transition is chosen by the controller. Different objectives for the controller give rise to different computational problems, aimed at computing optimal achievable objective values and optimal strategies. OC-MDPs are in fact equivalent to a controlled extension of (discrete-time) Quasi-Birth-Death processes (QBDs), a purely stochastic model heavily studied in queueing theory and applied probability. They can thus be viewed as a natural "adversarial" extension of a classic stochastic model. They can also be viewed as a natural probabilistic/controlled extension of classic onecounter automata. OC-MDPs also subsume (as a very restricted special case) a recently studied MDP model called "solvency games" that model a risk-averse gambling scenario. Basic computational questions for OC-MDPs include "termination" questions and "limit" questions, such as the following: does the controller have a strategy to ensure that the counter (which may, for example, count the number of jobs in the queue) will hit value 0 (the empty queue) almost surely (a.s.)? Or that the counter will have lim sup value ∞, a.s.? Or, that it will hit value 0 in a selected terminal state, a.s.? Or, in case such properties are not satisfied almost surely, compute their optimal probability over all strategies. We provide new upper and lower bounds on the complexity of such problems. Specifically, we show that several quantitative and almost-sure limit problems can be answered in polynomial time, and that almost-sure termination problems (without selection of desired terminal states) can also be answered in polynomial time. On the other hand, we show that the almost-sure termination problem with selected terminal states is PSPACE-hard and we provide an exponential time algorithm for this problem. We also characterize classes of strategies that suffice for optimality in several of these settings. Our upper bounds combine a number of techniques from the theory of MDP reward models, the theory of random walks, and a variety of automata-theoretic methods.given probability. Such questions are similar in spirit to questions asked in the rich literature on "adversarial queueing theory" (see, e.g., [4]), although this is a somewhat different setting. These considerations lead naturally to the extension of QBDs with control, and thus to OC-MDPs. Indeed, MDP variants of QBDs have already been studied in the stochastic modeling literature, see [27,19]. However, ...

Analyzing probabilistic pushdown automata

“…A formal definition of pOC adopted in this paper is consistent with the one used in recent works such as [11,15,10].…”

“…The underlying method is based on analyzing the trend, i.e., the long-run average change in counter value per transition. This is an important proof concept which turned out to be useful also in the more general setting of MDPs and stochastic games over one-counter automata (see, e.g., [11,9,10]). Therefore, we explain the main idea in greater detail.…”

Bisimulation equivalence and regularity for real-time one-counter automata

“…We can refer to the recent paper [33] and the references therein. Other problems studied for one-counter automata in the verification community can be exemplified by papers [34,35,36,37,38,39].…”

Games and Markov Decision Processes with Mean-Payoff Parity and Energy Parity Objectives

“…One classical example of quantitative objective is the mean-payoff objective [15,19]. Recently, the energy objectives (corresponding to total-payoff functions) have also been considered in the design of resource-constrained embedded systems such as power-limited systems [2,5], as well as in queueing processes, and gambling models (see also [3] and references therein). The energy objective requires that the sum of the rewards be always nonnegative along a trace.…”