Learning Policies for First Person Shooter Games Using Inverse Reinforcement Learning

Abstract

The creation of effective autonomous agents (bots) for combat scenarios has long been a goal of the gaming industry. However, a secondary consideration is whether the autonomous bots behave like human players; this is especially important for simulation/training applications which aim to instruct participants in real-world tasks. Bots often compensate for a lack of combat acumen with advantages such as accurate targeting, predefined navigational networks, and perfect world knowledge, which makes them challenging but often predictable opponents. In this paper, we examine the problem of teaching a bot to play like a human in first-person shooter game combat scenarios. Our bot learns attack, exploration and targeting policies from data collected from expert human player demonstrations in Unreal Tournament. We hypothesize that one key difference between human players and autonomous bots lies in the relative valuation of game states. To capture the internal model used by expert human players to evaluate the benefits of different actions, we use inverse reinforcement learning to learn rewards for different game states. We report the results of a human subjects' study evaluating the performance of bot policies learned from human demonstration against a set of standard bot policies. Our study reveals that human players found our bots to be significantly more human-like than the standard bots during play. Our technique represents a promising stepping-stone toward addressing challenges such as the Bot Turing Test (the CIG Bot 2K Competition).

Persona Agents from Playtraces and Emotion

“…This algorithm class allows for the generation of reward functions based on observed player traces. An example of this approach can be seen in the work by Tastan et al (Tastan and Sukthankar 2011). These authors propose an agent that generates policies for the competitive game Unreal Tournament.…”

POMCoP: Belief Space Planning for Sidekicks in Cooperative Games

“…In general, the problem of constructing a human model is challenging, particularly for domains where human strategies are unpredictable. One possible option would be to learn a model from observed human data, either online (Barrett, Stone, and Kraus 2011) or offline (Tastan and Sukthankar 2012;Orkin 2008;Broz, Nourbakhsh, and Simmons 2011). Alternatively a human could be modeled as a noisy optimal solver for an MDP or POMDP formulation of the game, assuming such policies could could be tractably found or approximated.…”

Assistant Agents for Sequential Planning Problems

“…If we have a set of observational data from humans playing the game, we can use machine learning to infer a policy and use it as a human model. This modeling approach has been demonstrated by researchers for a variety of machine learning techniques including decision trees (Barrett, Stone, and Kraus 2011) and reinforcement learning (Tastan and Sukthankar 2012). We will call this general approach the machine learning model.…”