Inducing Social Optimality in Games via Adaptive Incentive Design

Abstract

The Markov game framework is widely used to model interactions among agents with heterogeneous utilities in dynamic, uncertain, societal-scale systems. In these settings, agents typically operate in a decentralized manner due to privacy and scalability concerns, often without knowledge of others' strategies. Designing decentralized learning algorithms that provably converge to rational outcomes remains challenging, especially beyond Markov zero-sum and potential games, which do not fully capture the mixed cooperative-competitive nature of real-world interactions. Our paper focuses on designing decentralized learning algorithms for general-sum Markov games, aiming to provide guarantees of convergence to approximate Nash equilibria. We introduce a Markov Near-Potential Function (MNPF), and show that MNPF plays a central role in the analysis of convergence of an actor-critic-based decentralized learning dynamics to approximate Nash equilibria. Our analysis leverages the two-timescale nature of actor-critic algorithms, where Q-function updates occur faster than policy updates. This result is further strengthened under certain regularity conditions and when the set of Nash equilibria is finite. Our findings provide a new perspective on the analysis of decentralized learning in multi-agent systems, addressing the complexities of real-world interactions.

Achieving Hierarchy-Free Approximation for Bilevel Programs With Equilibrium Constraints

“…The question of inducing a desirable outcome in non-cooperative games can be traced back to the work of Pigou (1920) on welfare economics. Bilevel programming has long been recognized as the standard approach to such inquiries in operations research, and more recently in the ML community (Mguni et al, 2019;Zheng et al, 2020;Liu et al, 2022;Maheshwari et al, 2022). Our algorithms are focused on the applications pertinent to this question.…”

“…A typical example is a Stackelberg game concerning a leader who aims to induce a desirable outcome in an economic or social system comprised of many self-interested followers, who can be seen as playing a non-cooperative game that converges to a Nash equilibrium (Dafermos, 1973;Requate, 1993;Marcotte and Marquis, 1992;Labbé et al, 1998;Ehtamo et al, 2002). More recently, motivated by such applications, developing efficient algorithms for solving bilevel programs with equilibrium constraints has also emerged as an essential topic in machine learning (Mguni et al, 2019;Zheng et al, 2020;Liu et al, 2022;Maheshwari et al, 2022).…”