Optimal impulsive control of piecewise deterministic Markov processes

Abstract

In this paper, we study the infinite-horizon expected discounted continuous-time optimal control problem for Piecewise Deterministic Markov Processes with both impulsive and gradual (also called continuous) controls. The set of admissible control strategies is supposed to be formed by policies possibly randomized and depending on the past-history of the process. We assume that the gradual control acts on the jump intensity and on the transition measure, but not on the flow. The so-called Hamilton-Jacobi-Bellman (HJB) equation associated to this optimization problem is analyzed. We provide sufficient conditions for the existence of a solution to the HJB equation and show that the solution is in fact unique and coincides with the value function of the control problem. Moreover, the existence of an optimal control strategy is proven having the property to be stationary and non-randomized. ARTICLE HISTORY

Optimal impulse control of a SIR epidemic

“…On the other hand, if there is no drift and the trajectories are piecewise constant, the model is called a continuous‐time Markov decision process (CTMDP) . The impulse control means the following: At particular discrete time moments, the decision maker decides to intervene by instantaneously moving the process to some new point in the state space; that new point may be also random in the cases of CTMDP and PDMP . Then, restarting at this new point, the process runs until the next intervention and so on.…”

“…Sometimes, such control is called “singular control.” The goal is to minimize the total (expected) accumulated cost, which may be discounted or not . The popular method of attack to such problems is dynamic programming . In other works, versions of the Pontryagin maximum principle are used.…”

“…Impulse control of various dynamical systems attracts the attention of many researchers, eg, those in other works 8,9,[11][12][13][14][15][16][17][18][19][20][21][22][23][24] to mention the most relevant and the most recent works. The underlying system can be described in terms of ordinary 8,9,[11][12][13]20,[22][23][24] or stochastic 17,21 differential equations.…”

“…The underlying system can be described in terms of ordinary 8,9,[11][12][13]20,[22][23][24] or stochastic 17,21 differential equations. In other works, [14][15][16]19 along with the given deterministic drift, there are spontaneous (or natural) Markov jumps of the state. Such models are called piecewise deterministic Markov processes (PDMPs); the drift is usually described by a fixed flow.…”

“…18 The impulse control means the following: At particular discrete time moments, the decision maker decides to intervene by instantaneously moving the process to some new point in the state space; that new point may be also random in the cases of CTMDP and PDMP. 18,19 Then, restarting at this new point, the process runs until the next intervention and so on. Sometimes, such control is called "singular control."…”