Real World Games Look Like Spinning Tops

Czarnecki, Wojciech Marian; Gidel, Gauthier; Tracey, Brendan D.; Tuyls, Karl; Omidshafiei, Shayegan; Balduzzi, David; Jaderberg, Max

doi:10.48550/arxiv.2004.09468

Cited by 10 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Would the benefits of RAP disappear if a single adversary had the ability to represent mixed Nash (for example, by adding a source of randomness to the adversary state)? Another interesting question to ask is whether the minimax games described here satisfy the "games-of-skill" hypothesis [5] which would provide an optimization-based reason for including adversary populations.…”

Section: Discussionmentioning

confidence: 99%

“…The use of a single adversary in the robustness literature is in contrast to the multi-player game literature. In multi-player games, large sets of adversaries are used to ensure that an agent cannot easily be exploited [29,5,3]. Drawing inspiration from this literature, we introduce RAP (Robustness via Adversary Populations): a randomly initialized population of adversaries that we sample from at each rollout and train alongside the agent.…”

Section: Introductionmentioning

confidence: 99%

“…Alphastar, the grandmaster-level Starcraft bot, uses a population of "exploiter" agents that fine-tune against the bot to prevent it from developing exploitable strategies [29]. [5] establishes a set of sufficient geometric conditions on games under which the use of multiple adversaries will ensure gradual improvement in the strength of the agent policy. They empirically demonstrate that learning in games can often fail to converge without populations.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Robust Reinforcement Learning using Adversarial Populations

Vinitsky¹,

Du²,

Parvate³

et al. 2020

Preprint

View full text Add to dashboard Cite

Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution to a zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training a single RL agent against a single adversary. In this work, we demonstrate that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary; the resulting policy is highly exploitable by new adversaries. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training. We empirically validate across robotics benchmarks that the use of an adversarial population results in a more robust policy that also improves out-of-distribution generalization. Finally, we demonstrate that this approach provides comparable robustness and generalization as domain randomization on these benchmarks while avoiding a ubiquitous domain randomization failure mode. * Equal authorship Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Robust Reinforcement Learning using Adversarial Populations

Vinitsky¹,

Du²,

Parvate³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…An empirical game theory and α-rank approach focuses on 2-player zero-sum games, and analyses them in terms of the transitive and non-transitive relationships between sampled strategies at playing the game [55], [56]. It shares a perspective with this work of creating 'tools that enable the discovery of a topology over games', and in reducing the role of human-crafted dimensions, but using graph theory to analyse the results of agent match-ups.…”

Section: Properties Of Agentsmentioning

confidence: 99%

Visualizing Multiplayer Game Spaces

2022

View full text Add to dashboard Cite

We compare four different 'game-spaces' in terms of their usefulness in characterising multi-player tabletop games, with a particular interest in any underlying change to a game's characteristics as the number of players changes. In each case we take a 16-dimensional feature space, and reduce it to a 2dimensional visualizable landscape.We find that a space obtained from optimization of parameters in Monte Carlo Tree Search (MCTS) is the most directly interpretable to characterise our set of games in terms of the relative importance of imperfect information, adversarial opponents and reward sparsity. These results do not correlate with a space defined using attributes of the game-tree.This dimensionality reduction does not show any general effect as the number of players. We therefore consider the question using the original features to classify the games into two sets; those for which the characteristics of the game changes significantly as the number of players changes, and those for which there is no such effect.

show abstract

“…The central limitation of the Elo system is that it assumes transitivity. This is not necessarily the case, and in fact there are games -such as rock-paper-scissors -where the Elos assigned to each player are entirely uninformative [25]- [27].…”

Section: Ratings and Elomentioning

confidence: 99%

Scaling Scaling Laws with Board Games

Jones

2021

Preprint

View full text Add to dashboard Cite

The largest experiments in machine learning now require resources far beyond the budget of all but a few institutions. Fortunately, it has recently been shown that the results of these huge experiments can often be extrapolated from the results of a sequence of far smaller, cheaper experiments. In this work, we show that not only can the extrapolation be done based on the size of the model, but on the size of the problem as well. By conducting a sequence of experiments using AlphaZero and Hex, we show that the performance achievable with a fixed amount of compute degrades predictably as the game gets larger and harder. Along with our main result, we further show that the test-time and train-time compute available to an agent can be traded off while maintaining performance.

show abstract

Real World Games Look Like Spinning Tops

Cited by 10 publications

References 0 publications

Robust Reinforcement Learning using Adversarial Populations

Robust Reinforcement Learning using Adversarial Populations

Visualizing Multiplayer Game Spaces

Scaling Scaling Laws with Board Games

Contact Info

Product

Resources

About