Deriving Explicit Control Policies for Markov Decision Processes Using Symbolic Regression

Abstract

In this paper, we introduce a novel approach to optimizing the control of systems that can be modeled as Markov decision processes (MDPs) with a threshold-based optimal policy. Our method is based on a specific type of genetic program known as symbolic regression (SR). We present how the performance of this program can be greatly improved by taking into account the corresponding MDP framework in which we apply it.The proposed method has two main advantages: (1) it results in near-optimal decision policies, and (2) in contrast to other algorithms, it generates closed-form approximations. Obtaining an explicit expression for the decision policy gives the opportunity to conduct sensitivity analysis, and allows instant calculation of a new threshold function for any change in the parameters. We emphasize that the introduced technique is highly general and applicable to MDPs that have a threshold-based policy. Extensive experimentation demonstrates the usefulness of the method. CCS CONCEPTS• Mathematics of computing → Stochastic processes.

A data-driven approach to deriving closed-form approximations for queueing problems using genetic algorithms

“…The proposed SR-based approach is still in its infancy, but initial results are promising. In [4], it was shown that the approach can be used to obtain symbolic expressions for MDP value functions, and [2] gives an example where the approach is also useful to approximate optimal control policies in MDPs.…”

A novel use of value iteration for deriving bounds for threshold and switching curve optimal policies

“…In this article Hristov, Bhulai, van der Mei, and Bosman () (cf also Hristov, ), optimal threshold approximations for the M/M/1 queue with admission control have been derived using an evolutionary algorithm, called symbolic regression. Typically, rejection cost is used.…”