Variance-penalized Markov decision processes: dynamic programming and reinforcement learning techniques

Abstract

In control systems theory, the Markov decision process (MDP) is a widely used optimization model involving selection of the optimal action in each state visited by a discrete-event system driven by Markov chains. The classical MDP model is suitable for an agent/decisionmaker interested in maximizing expected revenues, but does not account for minimizing variability in the revenues. An MDP model in which the agent can maximize the revenues while simultaneously controlling the variance in the revenues is proposed. This work is rooted in machine learning/neural network concepts, where updating is based on system feedback and step sizes. First, a Bellman equation for the problem is proposed. Thereafter, convergent dynamic programming and reinforcement learning techniques for solving the MDP are provided along with encouraging numerical results on a small MDP and a preventive maintenance problem.

A unified algorithm framework for mean-variance optimization in discounted Markov decision processes

“…These problems include, but not limited to, the risk-averse variance and mean-variance optimizations in discounted and average MDPs, and these four metrics can be covered by ( 6). When the discount factor α ↑ 1, the problem turns into the mean-variance maximization in average MDPs (Xia 2020, Gosavi 2014) (see Remark 1). When the risk-aversion parameter β is large enough with respect to the mean, the problem degrades to the variance minimization problem (for average MDPs, see (Xia 2016)).…”

“…2. For the unified set of problems, a set of algorithms can be developed and analyzed in the framework, such as the policy gradients (Prashanth andGhavamzadeh 2013, Bisi et al 2020), the policy iterations (Xia 2016, 2020, Zhang et al 2021, and the value iteration (Gosavi 2014). The missing convergence analyses in some previous works can be developed as well.…”

“…With the ordinary differential equation approach, they prove the asymptotic local convergences of the algorithms. Gosavi (2014) proposes a model-free algorithm analogous to Q-learning for the mean-variance problem in average reinforcement learning (RL). The algorithm is validated with a numerical experiment, but the convergence analysis is missing.…”

Enhanced robot learning using fuzzy Q-Learning & context-aware middleware

“…Unlike DP, RL does not require an explicit model of the behavior of the system. This, however, does not imply that RL techniques cannot take advantage of a system behavior model [6], [1].…”

“…In motion planning however, the path has features that influence the optimal path to the end-point pose. The following equations ( 1,2) show the differences between Finite and Infinite Horizon spaces.…”