Actor-critic is implicitly biased towards high entropy optimal policies

Abstract

We show that the simplest actor-critic method -a linear softmax policy updated with TD through interaction with a linear MDP, but featuring no explicit regularization or explorationdoes not merely find an optimal policy, but moreover prefers high entropy optimal policies. To demonstrate the strength of this bias, the algorithm not only has no regularization, no projections, and no exploration like ǫ-greedy, but is moreover trained on a single trajectory with no resets. The key consequence of the high entropy bias is that uniform mixing assumptions on the MDP, which exist in some form in all prior work, can be dropped: the implicit regularization of the high entropy bias is enough to ensure that all chains mix and an optimal policy is reached with high probability. As auxiliary contributions, this work decouples concerns between the actor and critic by writing the actor update as an explicit mirror descent, provides tools to uniformly bound mixing times within KL balls of policy space, and provides a projection-free TD analysis with its own implicit bias which can be run from an unmixed starting distribution.

Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity

“…Note that the previous convergence recovers (7) when taking ρ = ν π * . The same procedure can also be directly applied to adapt the analysis of the SHPMD method.…”

“…Theorem 3.2 is also related to [7], which shows that the actor-critic method produces policy iterate that has bounded Kullback-Leibler (KL) divergence to the optimal policy with maximal entropy π * U . Our differences mainly exists in the following aspects: (1) We study general discounted MDP with finite state and action.…”

“…Our differences mainly exists in the following aspects: (1) We study general discounted MDP with finite state and action. On the other hand, [7] consider a slightly restrictive MDP class with a special linear transition kernel; (2) We establish exact convergence to the optimal policy with the maximal entropy π * U , instead of bounding the KL divergence KL(π k π * U ) as in [7]; We refer interested readers to [7] for an excellent discussion on the benefits of maintaining a high-entropy policy throughout policy optimization process.…”

A Novel Framework for Policy Mirror Descent with General Parametrization and Linear Convergence

“…We present the proof of Lemma 5.1 in Appendix B.1. Lemma 5.1 describes a relationship between any two policies and a policy belong to the Bregman projected policy class associated to F Θ and h. While similar results have been obtained and exploited for the tabular setting (Xiao, 2022) and for the negative entropy mirror map (Liu et al, 2019;Hu et al, 2021), Lemma 5.1 is the first to allow any parametrization class F Θ and any choice of mirror map. Since Lemma 5.1 does not depend on Algorithm 1, we expect it to be helpful in contexts outside this work.…”

“…for all s ∈ S. In this example, AMPO recovers tabular NPG (Shani et al, 2020) and NPG with log-linear polices (Hu et al, 2021) when f θ (s, a) = θ s,a and when f θ and Q t are linear functions for all t ≥ 0, respectively. We refer to Appendix A.1 for details and an extension to Tsallis entropy.…”