Regularized Gradient Descent Ascent for Two-Player Zero-Sum Markov Games

Abstract

We study the problem of finding the Nash equilibrium in a two-player zero-sum Markov game. Due to its formulation as a minimax optimization program, a natural approach to solve the problem is to perform gradient descent/ascent with respect to each player in an alternating fashion. However, due to the non-convexity/non-concavity of the underlying objective function, theoretical understandings of this method are limited. In our paper, we consider solving an entropy-regularized variant of the Markov game. The regularization introduces structure into the optimization landscape that make the solutions more identifiable and allow the problem to be solved more efficiently. Our main contribution is to show that under proper choices of the regularization parameter, the gradient descent ascent algorithm converges to the Nash equilibrium of the original unregularized problem. We explicitly characterize the finite-time performance of the last iterate of our algorithm, which vastly improves over the existing convergence bound of the gradient descent ascent algorithm without regularization. Finally, we complement the analysis with numerical simulations that illustrate the accelerated convergence of the algorithm.

Can We Find Nash Equilibria at a Linear Rate in Markov Games?

“…By choosing small regularization weights, their method can find an -Nash equilibrium in O(1/ ) iterations. Zeng et al (2022) also consider adding entropy regularization to help find Nash equilibria in zero-sum Markov games. They prove the O(t −1/3 ) convergence rate of a variant of GDA by driving regularization weights dynamically to zero.…”

“…However, due to the nonconvexity-nonconcavity, theoretical understanding of zero-sum Markov games is sparser. Existing methods have either sublinear rates for finding Nash equilibria, or linear rates for finding regularized Nash equiliria such as quantal response equilibria which are approximations for Nash equilibria Alacaoglu et al (2022); Cen et al (2021); Daskalakis et al (2020); Pattathil et al (2022); Perolat et al (2015); Wei et al (2021); Yang and Ma (2022); Zeng et al (2022); Zhang et al (2022); Zhao et al (2022). A natural question is:…”