Towards Autonomous Reinforcement Learning: Automatic Setting of Hyper-parameters using Bayesian Optimization

Abstract

With the increase of machine learning usage by industries and scientific communities in a variety of tasks such as text mining, image recognition and self-driving cars, automatic setting of hyper-parameter in learning algorithms is a key factor for obtaining good performances regardless of user expertise in the inner workings of the techniques and methodologies. In particular, for a reinforcement learning algorithm, the efficiency of an agent learning a control policy in an uncertain environment is heavily dependent on the hyper-parameters used to balance exploration with exploitation. In this work, an autonomous learning framework that integrates Bayesian optimization with Gaussian process regression to optimize the hyper-parameters of a reinforcement learning algorithm, is proposed. Also, a bandits-based approach to achieve a balance between computational costs and decreasing uncertainty about the \textit{Q}-values, is presented. A gridworld example is used to highlight how hyper-parameter configurations of a learning algorithm (SARSA) are iteratively improved based on two performance functions.

Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning

“…Other studies, on the other hand, focused on resorting to Bayesian optimization to optimize several RL hyper-parameters at the same time. In this setting, a preceding work is Barsce et al (2017) 44 , where a Bayesian optimization framework was proposed to optimize RL hyper-parameters. However, in such work, Bayesian optimization and the RL algorithm are decoupled in such a way that the meta-learning does not make specific assumptions with respect to an RL algorithm, making the method also inefficient because the learned tuples of experience ( , , , ′ ) are all aggregated in the metric selected for the objective function and, therefore, cannot be used directly to improve the meta-learning layer.…”

“…The reasoning behind integrating Bayesian optimization with an RL algorithm is that using a black-box approach is well suited for an expensive optimization task such as RL, by making the most of past queries in order to maximize the gain of selecting the next query of hyper-parameter setting through the acquisition function. As it employs Bayesian optimization for optimizing RL algorithms, this architecture has its foundations in RLOpt 44 .…”

Reinforcement learning for the traveling salesman problem with refueling

“…In RL, an agent learns from rewards and penalties in interacting with an environment [68]. One of the main topics of investigation in RL is the estimation of learning parameters, like learning rate (α) and discount factor (γ ), -greedy and reinforcement function [6,17,23,24,40,54,63]. In fact, parameter definition can directly influence a good route learning [5,12,52,54].…”

Improving the efficiency of reinforcement learning for a spacecraft powered descent with Q-learning

“…Automated hyperparameter selection is a very similar problem to meta-learning since it often uses a higher level learning procedure to "train" the hyperparameters of the lower level algorithm. These automated methods use a variety of intelligent approaches such as evolutionary computation (Schweighofer and Doya 2003;Young et al 2015) and Bayesian optimisation methods (Barsce et al 2017;Bergstra et al 2011).…”