On the sample complexity of actor-critic method for reinforcement learning with function approximation

Abstract

Reinforcement learning, mathematically described by Markov Decision Problems, may be approached either through dynamic programming or policy search. Actor-critic algorithms combine the merits of both approaches by alternating between steps to estimate the value function and policy gradient updates. Due to the fact that the updates exhibit correlated noise and biased gradient updates, only the asymptotic behavior of actor-critic is known by connecting its behavior to dynamical systems. This work puts forth a new variant of actor-critic that employs Monte Carlo rollouts during the policy search updates, which results in controllable bias that depends on the number of critic evaluations. As a result, we are able to provide for the first time the convergence rate of actor-critic algorithms when the policy search step employs policy gradient, agnostic to the choice of policy evaluation technique. In particular, we establish conditions under which the sample complexity is comparable to stochastic gradient method for non-convex problems or slower as a result of the critic estimation error, which is the main complexity bottleneck. These results hold in continuous state and action spaces with linear function approximation for the value function. We then specialize these conceptual results to the case where the critic is estimated by Temporal Difference, Gradient Temporal Difference, and Accelerated Gradient Temporal Difference. These learning rates are then corroborated on a navigation problem involving an obstacle and the pendulum problem which provide insight into the interplay between optimization and generalization in reinforcement learning.

Recent advances in reinforcement learning in finance

“…For example, Kumar et al. (2019) provided a convergence rate analysis for a nested‐loop Actor–Critic algorithm to the stationary point through quantifying the smallest number of actor updates k required to attain $$\inf _{0\le m\le k}\Vert \nabla J(\theta ^{(k)})\Vert ^2< \varepsilon$$. We denote this smallest number as K .…”

“…There are three main ways to execute the algorithm. In the nested‐loop setting (see, e.g., Kumar et al., 2019; Xu et al., 2020a), the actor updates the policy in the outer loop after the critic's repeated updates in the inner loop. The second way is the two time‐scale setting (see, e.g., Xu et al., 2020b), where the actor and the critic update their parameters simultaneously with different learning rates.…”

A Comprehensive Survey of Machine Learning Methodologies with Emphasis in Water Resources Management

“…Actor-critic methods [188] Actor-critic methods combine policy gradient and value function estimation. The actor learns the policy, while the critic estimates the value function to evaluate the policy's performance.…”

An Intelligent Control Method for Servo Motor Based on Reinforcement Learning

“…Compared with the on-policy algorithm, the off-policy algorithm can be used to explore the environment when collecting interactive data within the environment. Thus, it can improve the utilization efficiency of data and will not affect the performance of the final policy [21,22].…”