Evolution Strategies for Approximate Solution of Bayesian Games

Abstract

We address the problem of solving complex Bayesian games, characterized by high-dimensional type and action spaces, many (> 2) players, and general-sum payoffs. Our approach applies to symmetric one-shot Bayesian games, with no given analytic structure. We represent agent strategies in parametric form as neural networks, and apply natural evolution strategies (NES) [wierstra2014natural] for deep model optimization. For pure equilibrium computation, we formulate the problem as bi-level optimization, and employ NES in an iterative algorithm to implement both inner-loop best response optimization and outer-loop regret minimization. In simple games including first- and second-price auctions, it is capable of recovering known analytic solutions. For mixed equilibrium computation, we adopt an incremental strategy generation framework, with NES as strategy generator producing a finite sequence of approximate best-response strategies. We then calculate equilibria over this finite strategy set via a model-based optimization process. Both our pure and mixed equilibrium computation methods employ NES to efficiently search for strategies over the functional space, given only black-box simulation access to noisy payoff samples. We experimentally demonstrate the efficacy of all methods on two simultaneous sealed-bid auction games with distinct type distributions, and observe that the solutions exhibit qualitatively different behavior in these two environments.

Computing equilibria by minimizing exploitability with best-response ensembles

“…For example: "The best responses were computed by selecting the best point of a uniform discretization for the onedimensional problems and by using a mixed-integer linear programming reformulation for the Colonel Blotto games." Li and Wellman [2021] extended the double oracle algorithm to n-player general-sum continuous Bayesian games, formulating equilibrium computation as a bi-level optimization problem. They represent agents as neural networks and optimize them using natural evolution strategies (NES) [Wierstra et al, 2008[Wierstra et al, , 2014 for inner-loop best-response optimization and outer-loop regret minimization.…”

Towards multi‐agent reinforcement learning‐driven over‐the‐counter market simulations

“…Therefore, Bayesian Nash equilibria are Nash equilibria on a larger pure strategy space where policies have been augmented with the agent type. We refer to Li and Wellman (2021) for the definition of Bayesian games and Bayesian Nash equilibria. One difference between Bayesian games and our game is that we do not assume that agents know the supertype profile $$\bm {\Lambda ^{LP}}$$, called the “type prior” in Bayesian games and assumed to be of common knowledge.…”