Policy derivation methods for critic-only reinforcement learning in continuous spaces

Abstract

This paper addresses the problem of deriving a policy from the value function in the context of critic-only reinforcement learning (RL) in continuous state and action spaces. With continuous-valued states, RL algorithms have to rely on a numerical approximator to represent the value function. Numerical approximation due to its nature virtually always exhibits artifacts which damage the overall performance of the controlled system. In addition, when continuousvalued action is used, the most common approach is to discretize the action space and exhaustively search for the action that maximizes the right-hand side of the Bellman equation. Such a policy derivation procedure is computationally involved and results in steady-state error due to the lack of continuity. In this work, we propose policy derivation methods which alleviate the above problems, by means of action space refinement, continuous approximation, and post-processing of the V-function by using symbolic regression. The proposed methods are tested on non-linear control problems: 1-DOF and 2-DOF pendulum swing-up problems, and on magnetic manipulation. The results show significantly improved performance in terms of cumulative return and computational complexity.

Symbolic regression driven by training data and prior knowledge

“…We use data measured on a real system and an empirical modelf (x) = −ic 1 x/(x 2 + c 2 ) 3 proposed in the literature [15] as the reference model. Parameters c 1 and c 2 were found empirically for the given system and this model was used to design well-performing nonlinear controllers in [1,8]. For this example, we define y = f (x).…”

“…We chose for this variant of GP since it has recently been shown in [3,4,18,24] that GP methods evolving this kind of compound regression models outperform conventional GP evolving a singletree structure representing the whole model. In particular, the base SNGP has been successfully used for several SR tasks from the reinforcement learning and robotics domains [1,2,12,13]. A detailed description of the base SNGP is beyond the scope of this paper.…”

“…From now on, we will use the term population in the sense of the population of models. The proposed algorithm is based on the NSGA-II algorithm [10] that uses the following domination principle: A solution x (1) is said to dominate another solution x (2) , if x (1) is not worse than x (2) in any objective and x (1) is strictly better than x (2) in at least one objective.…”

“…Symbolic regression (SR) is an approach that generates models in the form of analytic equations that can be constructed by using even very small training data sets. SR has been used in nonlinear data-driven modeling with quite impressive results [1,2,12,13,23,25,26].…”

Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis

“…In addition to leverage genetic programming to generate interpretable policies, the influence of the smooth property of symbolic approximation function in reinforcement learning has also been studied. References [1] and [2] proposed a symbolic regression method to obtain a smooth v-function from the original value function, and experiment results reveal that policy derived from the smooth value function performs much better than the policy derived from the original value function.…”