Towards autonomous reinforcement learning: Automatic setting of hyper-parameters using Bayesian optimization

Abstract

With the increase of machine learning usage by industries and scientific communities in a variety of tasks such as text mining, image recognition and self-driving cars, automatic setting of hyper-parameter in learning algorithms is a key factor for achieving satisfactory performance regardless of user expertise in the inner workings of the techniques and methodologies. In particular, for a reinforcement learning algorithm, the efficiency of an agent learning a control policy in an uncertain environment is heavily dependent on the hyper-parameters used to balance exploration with exploitation. In this work, an autonomous learning framework that integrates Bayesian optimization with Gaussian process regression to optimize the hyper-parameters of a reinforcement learning algorithm, is proposed. Also, a bandits-based approach to achieve a balance between computational costs and decreasing uncertainty about the Q-values, is presented. A gridworld example is used to highlight how hyper-parameter configurations of a learning algorithm (SARSA) are iteratively improved based on two performance functions.

An Automated Deep Reinforcement Learning Pipeline for Dynamic Pricing

“…One popular way of automating algorithm selection is modeling the problem as MAB problem and assigning an action to each algorithm [26]. BO, which works well for automated machine learning, has been used for hyperparameters tuning for RL algorithms [27] and for adjusting weights of different objectives in the reward function [28]. In [29], hyper-parameters of the RL algorithm and the network structure are jointly optimized using the Genetic algorithm in which each individual is a DRL agent.…”

Vulnerable Road Users and Connected Autonomous Vehicles Interaction: A Survey

“…The algorithms focus on establishing a method to define the relationship between a set of variables (which represent the characteristics) and a continuous target variable. Examples of such algorithms being applied in self-driving systems include Bayesian regression [71], neural network regression [72] and decision forest regression [73].…”

Aplicação de Aprendizado por Reforço no Blackjack: Estudo de Caso de Estimação de Parâmetros

“…aprendizado (0 < α ≤ 1) regula a velocidade em que as novas informações sobrepõem-se sobre o aprendizado já armazenado na matriz Q. Já o fator de desconto tem o papel de controlar a influência das recompensas futuras: se γ = 0, o reforço imediato tem grande influência; se 0 < γ ≤ 1, as recompensas futuras são descontadas; se γ = 1, as recompensas não são descontadas. Nesse aspecto, um desafio do ARé configurar os parâmetros de aprendizado de forma a maximizar o desempenho no domínio em estudo, pois , α e γ podem assumir diferentes combinações de valores (Schweighofer and Doya, 2003;Even-Dar and Mansour, 2003;Barsce et al, 2017;Ottoni et al, 2018).…”

“…De fato, um dos principais aspectos do ML e também do ARé a estimação de parâmetros, como taxa de aprendi-zado, fator de desconto, − greedy e função de reforço (Schweighofer and Doya, 2003;Even-Dar and Mansour, 2003;Barsce et al, 2017;Ottoni et al, 2018;Liessner et al, 2019;Hutter et al, 2019). Nesse aspecto, o desafioé propor métodos para a otimização e recomendação de parâmetros, de modo a otimizar o desempenho do aprendizado (Hutter et al, 2019).…”