Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing

Abstract

Deep Reinforcement Learning (RL) is proven powerful for decision making in simulated environments. However, training deep RL model is challenging in real world applications such as production-scale health-care or recommender systems because of the expensiveness of interaction and limitation of budget at deployment. One aspect of the data inefficiency comes from the expensive hyper-parameter tuning when optimizing deep neural networks. We propose Adaptive Behavior Policy Sharing (ABPS), a data-efficient training algorithm that allows sharing of experience collected by behavior policy that is adaptively selected from a pool of agents trained with an ensemble of hyper-parameters. We further extend ABPS to evolve hyper-parameters during training by hybridizing ABPS with an adapted version of Population Based Training (ABPS-PBT). We conduct experiments with multiple Atari games with up to 16 hyper-parameter/architecture setups. ABPS achieves superior overall performance, reduced variance on top 25% agents, and equivalent performance on the best agent compared to conventional hyperparameter tuning with independent training, even though ABPS only requires the same number of environmental interactions as training a single agent. We also show that ABPS-PBT further improves the convergence speed and reduces the variance.Recent years have witnessed the success of deep reinforcement learning (RL) in solving complex sequential decision making problems such as games [Mnih et al., 2013, Schulman et al., 2015, Mnih et al., 2016. However, it is yet proven to be effective in real world applications such as large-scale ˚work was done while an intern in Google

Unentangled quantum reinforcement learning agents in the OpenAI Gym

“…The difficulty comes from the fact that the state-action space is exponentially large [9], where the optimal policy can not be explored efficiently. A mainstream strategy is to feed numerous trials to RL models in the optimization process to enhance performance [10][11][12][13][14][15][16][17]. Nevertheless, such a sample inefficiency challenges the applicability of RL towards large-scale problems [8], where the requested computational overhead is expensive or even unaffordable.…”