On gradual-impulse control of continuous-time Markov decision processes with exponential utility

Abstract

We consider a gradual-impulse control problem of continuous-time Markov decision processes, where the system performance is measured by the expectation of the exponential utility of the total cost. We show, under natural conditions on the system primitives, the existence of a deterministic stationary optimal policy out of a more general class of policies that allow multiple simultaneous impulses, randomized selection of impulses with random effects, and accumulation of jumps. After characterizing the value function using the optimality equation, we reduce the gradual-impulse control problem to an equivalent simple discrete-time Markov decision process, whose action space is the union of the sets of gradual and impulsive actions.

Counterexample Generation for Probabilistic Model Checking Micro-Scale Cyber-Physical Systems

“…In the future, we will continue to optimize HGA to generate counterexamples for MCPSs against requirements in PCTL* which is a super set of PCTL. At the same time, we would like to extend HGA to generate counterexamples for MCPSs models with continuous-time or hybrid-time stochastic behaviors, such as CTMDPs (Continuoustime Markov Decision Processes), PHA (Probabilistic Hybrid Automata), against CSL (Continuous-time Stochastic Logic) stochastic communication logics [44,45]. Funding: This research was funded by the Singapore-UK Cyber Security of EPSRC under Grant Nos.…”

Gradual-Impulsive Control for Continuous-Time Markov Decision Processes with Total Undiscounted Costs and Constraints: Linear Programming Approach via a Reduction Method

“…Pure stationary strategies are the easiest for implementation. They are often sufficient for problems with a single objective, i.e., when J = 0 (see e.g., [9,17]) or when the model satisfies additional structural conditions, such as being convex or atomless (see e.g., [11,14]). The next example demonstrates that without further conditions, pure stationary strategies are often not sufficient for the constrained problem (2.3).…”