Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach

Abstract

International audienc

Optimal impulsive control of piecewise deterministic Markov processes

“…(c) One can say that a finite simultaneous sequence of impulsive actions is one intervention and C i (x, a) > δ > 0. This approach, demonstrated in [8,9], causes no principal difficulties, but leads to cumbersome notations. Suppose a strategy u = (u n ) n∈N ∈ U is fixed with u n = ψ n , π n , γ 0 n , γ 1 n for n ∈ N * and let n ∈ N * be fixed.…”

“…If the flow is constant then the model transforms into the standard continuous-time Markov decision process. Both the gradual and impulsive controls were studied in [8,9] where more references can be found.…”

Gradual-impulsive control for continuous-time Markov decision processes with total undiscounted costs and constraints: linear programming approach via a reduction method

“…The optimal stopping problem is an important example of impulsive control models, where the decision maker can decide when to stop the process, applying the impulse once and for all, see e.g., [1,3]. Compared to the aforementioned two sub-models, there is relatively less literature on gradual-impulsive CTMDP models, see e.g., [10,11,22,25,27,28].…”

“…In the gradual-impulsive CTMDP model considered in this paper an impulse can be applied at any time moment, and one can apply multiple impulses at a single time moment. Such gradual-impulsive CTMDP models were considered in [27,28] and more recently in [10,11]. In [28], which is a refinement of [27], and in [10,11], the authors handled multiple simultaneous impulses by extending the time or the state suitably, and after that, developed a theory for the resulting new point process, which is more complicated than the uncontrolled version of the original process.…”

On the computation of Whittle’s index for Markovian restless bandits

“…However, they do appear naturally in several application domains such as epidemiology, reliability theory, inventory control, and congestion control in communication networks. We refer the interested reader to Dufour and Piunovskiy [2016] for a brief survey on MDPs with impulsive control. In order to calculate Whittle's index in the case of infinite rates, we reformulate the problem with impulse cost to a problem with only cost per unit of time.…”