Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding

Abstract

We study the problem of efficiently generating high-quality and diverse content in games. Previous work on automated deckbuilding in Hearthstone shows that the quality diversity algorithm MAP-Elites can generate a collection of high-performing decks with diverse strategic gameplay. However, MAP-Elites requires a large number of expensive evaluations to discover the diverse collection of decks. We propose assisting MAP-Elites with a deep surrogate model trained online to predict game outcomes with respect to candidate decks. MAP-Elites discovers a diverse dataset to improve the surrogate model accuracy, while the surrogate model helps guide MAP-Elites towards promising new content. In a Hearthstone deckbuilding case study, we show that our approach improves the sample efficiency of MAP-Elites and outperforms a model trained offline with random decks, as well as a linear surrogate model baseline, setting a new state-of-the-art for quality diversity approaches in the application domain of automated Hearthstone deckbuilding.

Covariance Matrix Adaptation MAP-Annealing

“…Several works combine gradient information with quality diversity optimization in ways that do not leverage the objective and measure gradients directly. For example, in model-based quality diversity optimization [26,31,6,46,54,78,27], prior work [66] trains an autoencoder on the archive of solutions and leverages the Jacobian of the decoder network to compute the covariance of the Gaussian perturbation. In quality diversity reinforcement learning (QD-RL), several works [60,62,57,75] approximate a reward gradient or diversity gradient via a critic network, action space noise, or evolution strategies and incorporate those gradients into a QD-RL algorithm.…”

“…Quality diversity optimization is a rapidly growing branch of stochastic optimization with applications in generative design [32,27,26], automatic scenario generation in robotics [21,20,19], reinforcement learning [60,62,57,75], damage recovery in robotics [14], and procedural content generation [30,24,78,15,48,73,68,67]. Our paper introduces a new quality diversity algorithm, CMA-MAE, that bridges the gap between single-objective optimization and quality diversity optimization.…”

A discretization-free metric for assessing quality diversity algorithms

“…Surrogate-assisted approaches have shown promise in accelerating problems which are both high-dimensional and expensive [25], [26].…”