Qualitative Controller Synthesis for Consumption Markov Decision Processes

Abstract

Consumption Markov Decision Processes (CMDPs) are probabilistic decision-making models of resource-constrained systems. In a CMDP, the controller possesses a certain amount of a critical resource, such as electric power. Each action of the controller can consume some amount of the resource. Resource replenishment is only possible in special reload states, in which the resource level can be reloaded up to the full capacity of the system. The task of the controller is to prevent resource exhaustion, i.e. ensure that the available amount of the resource stays non-negative, while ensuring an additional linear-time property. We study the complexity of strategy synthesis in consumption MDPs with almost-sure Büchi objectives. We show that the problem can be solved in polynomial time. We implement our algorithm and show that it can efficiently solve CMDPs modelling real-world scenarios.

Polynomial-Time Algorithms for Multi-Agent Minimal-Capacity Planning

“…In each iteration of the binary search, we check whether there exists some strategy that almost surely reaches t from s with the current capacity. This process requires at most log(c) instances of the polynomial algorithms of [12].…”

“…Here, the primary objective is to ensure that the system does not run out of resources during its operation. Energy and consumption MDPs can model systems operating in stochastic environments under resource constraints, with the latter admitting polynomial-time algorithms for qualitative planning [12].…”

“…The proposed reduction-based solution requires a graph with target states as vertices and where the cost of an edge (t 1 , t 2 ) represents the minimal capacity needed by an agent to almostsurely reach t 2 from t 1 . Qualitative strategy synthesis in consumption MDPs can be performed in polynomial-time with respect to the model size [12] and the minimal capacity for each edge can be precisely computed using binary search with logarithmic number of computations in capacity. Therefore, the reduction is polynomial.…”

“…However, planning in energy MDPs [20] is at least as hard as solving mean-payoff graph games [19], which makes the existence of a polynomial-time algorithm unlikely. Finally, in the recently introduced consumption MDPs [12], the agents are restricted by finite capacity, transitions can only decrease the resource level, and agents can replenish the resource only in a set of designated reload states (to full capacity only). These restrictions are sufficient to admit polynomial-time algorithms for qualitative planning (or solving consumption games [21]).…”

Efficient Strategy Synthesis for MDPs with Resource Constraints

“…We first introduced CMDPs and presented the algorithm for the Büchi objective in [15]. In contrast to [15], this paper contains the omitted proofs, it extends the algorithmic core with the reachability objective and it introduces goal-leaning and threshold heuristics that attempt to improve expected reachability time of targets.…”