Optimal control of time-inhomogeneous Markov chains with application to dam management

Abstract

We consider a time-inhomogeneous Markov chain with a compound Poisson process as an input as an important approximation to get the solution of real problems. Outflows are comprised of both controlled and uncontrolled counting processes. We demonstrate the utility of this model in the specific problem of demand control and flood prevention in a nearly full dam, where the dam is modeled as a continuous-time controllable Markov chain under control resource constraints. This work significantly extends previous results because the inflow of rain is potentially sudden and of large volume, so a compound Poisson process is preferred to simple counting processes. On the other hand the safe release of water is constrained and so is modelled as a simple counting process. A proof of the infinitesimal generator of the Markov chain with these characteristics is given and a numerical example demonstrates the effectiveness of the controls.

Towards Tensor Representation of Controlled Coupled Markov Chains

“…where we used the definition of tensor sum (5), the representation of the inner product as a component-wise sum, and denoted…”

“…The analysis and optimization of systems of coupled Markov Chains (MC) appear in various applied areas such as environmental management [1,2], control of dams [3][4][5], control of data transmission systems for the avoidance of congestion [6][7][8], and many others. In general, the optimal control of MC with constraints and various criteria leads to the solution of a system of ordinary differential equations.…”