Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality

Abstract

Designing off-policy reinforcement learning algorithms is typically a very challenging task, because a desirable iteration update often involves an expectation over an on-policy distribution. Prior off-policy actor-critic (AC) algorithms have introduced a new critic that uses the density ratio for adjusting the distribution mismatch in order to stabilize the convergence, but at the cost of potentially introducing high biases due to the estimation errors of both the density ratio and value function. In this paper, we develop a doubly robust off-policy AC (DR-Off-PAC) for discounted MDP, which can take advantage of learned nuisance functions to reduce estimation errors. Moreover, DR-Off-PAC adopts a single timescale structure, in which both actor and critics are updated simultaneously with constant stepsize, and is thus more sample efficient than prior algorithms that adopt either two timescale or nested-loop structure. We study the finite-time convergence rate and characterize the sample complexity for DR-Off-PAC to attain an ǫ-accurate optimal policy. We also show that the overall convergence of DR-Off-PAC is doubly robust to the approximation errors that depend only on the expressive power of approximation functions. To the best of our knowledge, our study establishes the first overall sample complexity analysis for a single time-scale off-policy AC algorithm.

Recent Advances in Reinforcement Learning in Finance

“…[229] gave the first non-asymptotic convergence guarantee for the two time-scale natural Actor-Critic algorithms with mean-squared sample complexity of order O( 1(1−γ) 9 ε 4 ). For single-scale Actor-Critic methods, the global convergence with sublinear convergence rate was established in both [74] and [230]. The non-asymptotic convergence of policy-based algorithms is shown in other settings, see [239] for the regret analysis of the REINFORCE algorithm for discounted MDPs and [7,146] for policy gradient methods in the setting of known model parameters.…”

“…In the third way, the single-scale setting, the actor and the critic update their parameters simultaneously but with a much larger learning rate for the actor than for the critic (see, e.g. [74,230]).…”

Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch

“…With the help of the Fisher information, the optimality of natural actor critic (Kakade, 2001;Peters and Schaal, 2008;Bhatnagar et al, 2009) is also established in both on-policy settings (Agarwal et al, 2020;Wang et al, 2019;Liu et al, 2020;Khodadadian et al, 2021b) and off-policy settings (Khodadadian et al, 2021a;Chen et al, 2021a). Moreover, Xu et al (2021) establish the convergence to stationary points of an off-policy actor critic with density ratio correction and a fixed sampling distribution. To study the optimality of the stationary points, Xu et al (2021) also make some assumptions about the Fisher information.…”

“…the ratio between the state distribution of the target policy and that of the behavior policy (Hallak and Mannor, 2017;Gelada and Bellemare, 2019;Liu et al, 2018;Nachum et al, 2019;Zhang et al, 2020b), can be used to correct the state distribution mismatch between the behavior policy and the target policy. Consequently, convergence to stationary points of actor-critic methods in off-policy settings with density ratio has also been established (Liu et al, 2019;Zhang et al, 2020c;Huang and Jiang, 2021;Xu et al, 2021).…”

“…Thanks to ρ t and t , in the limit, ( 5) is still a stochastic gradient ascent algorithm following the gradient ∇J(π θt ; p 0 ) if q t converges to q π θ t . Theoretical analysis of variants of (5) includes Liu et al (2019); Zhang et al (2020c); Huang and Jiang (2021); Xu et al (2021). Practitioners, however, usually use only ρ t but completely ignore t , yielding variants of…”