Evolving opponent models for Texas Hold 'Em

Abstract

Abstract-Opponent models allow software agents to assess a multi-agent environment more accurately and therefore improve the agent's performance. This paper makes use of coarse approximations to game-theoretic player representations to improve the performance of software players in Limit Texas Hold 'Em poker. A 10-parameter model, intended to model a combination, or mixture, of various strategies is developed to represent the opponent. A 'mixture identifier' is then evolved using the NEAT neuroevolution method to estimate values of these parameters for arbitrary opponents. To evaluate this approach, two poker players, represented as neural networks, were evolved under the same conditions, one with the mixture identifier, and one without. The player trained with access to the identifier achieved consistently higher and more stable fitness during evolution compared with the player without the identifier. Further, the player with the identifier outplays the other in a heads-up match after training, winning on average 60% of the money at the table. These results demonstrate that opponent modeling is effective even with low-dimensional models and conveys an advantage to players trained to use these models.

Neuroevolution in Games: State of the Art and Open Challenges

“…Neuroevolution can be used in the specific role of predicting opponent strategy, as part of a player whose other parts might or might not be based on neuroevolution. Lockett and Miikkulainen evolved networks that could predict the other player's strategy in Texas Hold'em Poker, increasing the win rate of agents that used the model [66].…”

Synthetic learning agents in game-playing social environments

“…Texas Hold ‘Em poker is a classic game used for studying learning mechanisms, especially for trying to predict opponents’ strategies and to play based on their model (Billings, Papp, Schaeffer, & Szafron, 1998; Ganzfried & Sandholm, 2011; Lockett & Miikkulainen, 2008). Lockett and Miikulainen (2008) used a coarse approximation to game-theoretic agents representations to improve the performance of agents in playing a Limit Texas Hold ‘Em poker by using opponent models initialized with a diverse mix of several parameters in order to develop agents using different strategies. The Deviation-Based Best Response (DBBR) algorithm has been developed by Ganzfriend and Sandholm (2011), for opponent modelling in large extensive-form games of imperfect information.…”

A Simulation Study of Texas Hold ’em Poker: What Taylor Swift Understands and James Bond Doesn’t

“…Perhaps the most interesting of these look at neuroevolution models, which are trained on opponents' behaviour on prior hands to predict when certain players might be playing suboptimally [4,8]. Impressively, even for a low-dimensional parameter space, some of these models have claimed on average 60% of table winnings when tested for a two-player game [9]. In addition to these quantitative studies, a substantial volume of work has been published in popular psychology [11,12], although it remains unclear whether many of the qualitative arguments are supported by quantitative evidence.…”