On the complexity of computing Markov perfect equilibrium in general-sum stochastic games

Abstract

Similar to the role of Markov decision processes in reinforcement learning, Markov games (also called stochastic games) lay down the foundation for the study of multi-agent reinforcement learning and sequential agent interactions. We introduce approximate Markov perfect equilibrium as a solution to the computational problem of finite-state stochastic games repeated in the infinite horizon and prove its PPAD-completeness. This solution concept preserves the Markov perfect property and opens up the possibility for the success of multi-agent reinforcement learning algorithms on static two-player games to be extended to multi-agent dynamic games, expanding the reign of the PPAD-complete class.

Semantic and Effective Communication for Remote Control Tasks with Dynamic Feature Compression

“…This design involves a trade-off between the minimization of the communication cost and the quality of control, which becomes less accurate as the transmission rate is reduced, in both the push-and pull-based versions. Using this model, we analyze the advantages and drawbacks of each configuration, proving relevant results and showing that the push-based system, while having better performance at the optimum, is a PPAD-hard problem [15].…”

“…However, reaching an NE is not a guarantee of Pareto optimality: games may have multiple NEs, and finding the optimal one is PPAD-hard [15]. The push-based approach may be actively harmful, even with respect to an AoI policy.…”

“…As finding the optimal solution to a Markov game is PPADhard [15], no polynomial-time algorithm can reliably find π * B,push . We can then give a counterexample to prove the second part of the theorem: we consider a simple MDP with 5 states and 2 actions, whose evolution is depicted in Fig.…”