Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient

Abstract

Deep Q-learning algorithms often suffer from poor gradient estimations with an excessive variance, resulting in unstable training and poor sampling efficiency. Stochastic variancereduced gradient methods such as SVRG have been applied to reduce the estimation variance . However, due to the online instance generation nature of reinforcement learning, directly applying SVRG to deep Q-learning is facing the problem of the inaccurate estimation of the anchor points, which dramatically limits the potentials of SVRG. To address this issue and inspired by the recursive gradient variance reduction algorithm SARAH (Nguyen et al. 2017), this paper proposes to introduce the recursive framework for updating the stochastic gradient estimates in deep Q-learning, achieving a novel algorithm called SRG-DQN. Unlike the SVRG-based algorithms, SRG-DQN designs a recursive update of the stochastic gradient estimate. The parameter update is along an accumulated direction using the past stochastic gradient information, and therefore can get rid of the estimation of the full gradients as the anchors. Additionally, SRG-DQN involves the Adam process for further accelerating the training process. Theoretical analysis and the experimental results on well-known reinforcement learning tasks demonstrate the efficiency and effectiveness of the proposed SRG-DQN algorithm.

A Deep Reinforcement Learning-Based Decision Support System for Automated Stock Market Trading

“…Multiple training attempts with the same beginning conditions will invariably result in somewhat different trading strategies with various results. This is also observed in the research study of Thibaut et al [68] and this is also discussed by Jia et al [69]. Hence it is necessary to validate the model with multiple trials and report the average.…”