<i>Pac-Man</i> Conquers Academia: Two Decades of Research Using a Classic Arcade Game

Rohlfshagen, Philipp; Liu, Jialin; Peŕez-Liebana, Diego; Lucas, Simon M.

doi:10.1109/tg.2017.2737145

Cited by 32 publications

(20 citation statements)

References 85 publications

(158 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…2 Choose non-negative values for the agent risk-taking parameters ϑt. 3 Initialize the action-value function q(a, s) and hence the policy π(a, s), s ∈ S, a ∈ A. 4 for t = 0, 1, 2, .…”

Section: Finding Policies Using Information Valuementioning

confidence: 99%

See 1 more Smart Citation

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Sledge

Prı́ncipe

2019

IEEE Trans. Games

View full text Add to dashboard Cite

Conventional reinforcement learning methods for Markov decision processes rely on weakly-guided, stochastic searches to drive the learning process. It can therefore be difficult to predict what agent behaviors might emerge. In this paper, we consider an information-theoretic cost function for performing constrained stochastic searches that promote the formation of risk-averse to risk-favoring behaviors. This cost function is the value of information, which provides the optimal trade-off between the expected return of a policy and the policy's complexity; policy complexity is measured by number of bits and controlled by a single hyperparameter on the cost function. As the policy complexity is reduced, the agents will increasingly eschew risky actions. This reduces the potential for high accrued rewards. As the policy complexity increases, the agents will take actions, regardless of the risk, that can raise the long-term rewards. The obtainable reward depends on a single, tunable hyperparameter that regulates the degree of policy complexity.We evaluate the performance of value-of-information-based policies on a stochastic version of Ms. Pac-Man. A major component of this paper is the demonstration that ranges of policy complexity values yield different game-play styles and explaining why this occurs. We also show that our reinforcementlearning search mechanism is more efficient than the others we utilize. This result implies that the value of information theory is appropriate for framing the exploitation-exploration trade-off in reinforcement learning.Index Terms-Value of information, constrained search, reinforcement learning, information theory Isaac J. Sledge is with the 2 objective of the agent is to clear the environment of pellets while navigating around the ghosts. However, after activating certain power-ups, the ghosts become vulnerable for a brief period of time. The agent can consume these ghosts for a score boost.The switch in ghost dynamics necessitates a change in the game-play strategy, since multiple distinct modes of behavior are required under different conditions [4,5]. Despite the need for multi-modal behaviors, conventional reinforcement-learning approaches have focused on constructing monolithic policies. Such policies would implement the same agent behaviors regardless of the vulnerability of the ghosts. Although it is possible to represent multimodal behavior with these policies, it can be difficult to learn such behavior. This is, in part, due to risk. For instance, throughout the learning process, an agent may have learned to avoid colliding with the ghosts. Without straying from this behavior, the agent will not learn that there are instances where it can safely chase the ghosts.In this paper, we consider an information-theoretic learning [6] approach for performing constrained stochastic searches that promote a continuum of risk-averse to risk-favoring agent behaviors during reinforcement learning. This, in turn, leads to a principled exploration of the state-action space that aids in the...

show abstract

“…2 Choose non-negative values for the agent risk-taking parameters ϑt. 3 Initialize the action-value function q(a, s) and hence the policy π(a, s), s ∈ S, a ∈ A. 4 for t = 0, 1, 2, .…”

Section: Finding Policies Using Information Valuementioning

confidence: 99%

“…Both of these issues can be illustrated with the arcade game Ms. Pac-Man [3]. Ms. Pac-Man is a predator-prey scenario set in a maze-like environment.…”

Section: Introductionmentioning

confidence: 99%

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Sledge

Prı́ncipe

2019

IEEE Trans. Games

View full text Add to dashboard Cite

show abstract

“…Different competitions address various AI/CI problems through different game frameworks, being IEEE's Conference on Computational Intelligence in Games one the events in which most of them take place on a yearly basis [11]. The Ms Pacman competitions [12] have been running for more than 10 years now, asking participants to write controllers for both Pacman and the ghosts using the Ms Pacman framework (an open implementation of the sequel to the original Pacman). The Geometry Friends competitions [13] asks for controllers that must succeed at playing levels in a custom cooperative video game.…”

Section: Related Workmentioning

confidence: 99%

Dota 2 Bot Competition

Font

Mahlmann

2019

IEEE Trans. Games

View full text Add to dashboard Cite

Multiplayer Online Battle Area (MOBA) games are a recent huge success both in the video game industry and the international eSports scene. These games encourage team coordination and cooperation, short and long-term planning, within a real-time combined action and strategy gameplay. Artificial Intelligence and Computational Intelligence in Games research competitions offer a wide variety of challenges regarding the study and application of AI techniques to different game genres. These events are widely accepted by the AI/CI community as a sort of AI benchmarking that strongly influences many other research areas in the field. This paper presents and describes in detail the Dota 2 Bot competition and the Dota 2 AI framework that supports it. This challenge aims to join both, MOBAs and AI/CI game competitions, inviting participants to submit AI controllers for the successful MOBA Defense of the Ancients 2 (Dota 2) to play in 1v1 matches, which aims for fostering research on AI techniques for real-time games. The Dota 2 AI framework makes use of the actual Dota 2 game modding capabilities to enable to connect external AI controllers to actual Dota 2 game matches using the original Free-to-Play game.

show abstract

“…Pac-Man and Ms. Pac-Man have been the focus of much research [5], the most relevant of which is presented below.…”

Section: Previous Ms Pac-man Researchmentioning

confidence: 99%

Neuroevolution of Multimodal Ms. Pac-Man Controllers Under Partially Observable Conditions

Price

Schrum

2019

2019 IEEE Congress on Evolutionary Computation (CEC)

View full text Add to dashboard Cite

Ms. Pac-Man is a well-known video game used extensively in AI research. Past research has focused on the standard, fully observable version of Ms. Pac-Man. Recently, a partially observable variant of the game has been used in the MS. PAC-MAN VS. GHOST TEAM COMPETITION at the Computational Intelligence and Games (CIG) conference. Restricting Ms. Pac-Man's view makes the game more challenging. Ms. Pac-Man can only see down halls within her direct line of sight. The approach to this domain presented in this paper extends an earlier approach using MM-NEAT, an algorithm for evolving modular neural networks. Experiments using several forms of evolved and human-specified modularity are presented. The best evolved agent uses a human-specified task division with output modules for different situations: no ghosts, edible ghosts, and threat ghosts. This approach placed first at the MS. PAC-MAN VS. GHOST TEAM COMPETITION at CIG 2018 against seven other competitors with an average score of 7736.63.

show abstract

Pac-Man Conquers Academia: Two Decades of Research Using a Classic Arcade Game

Cited by 32 publications

References 85 publications

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Dota 2 Bot Competition

Neuroevolution of Multimodal Ms. Pac-Man Controllers Under Partially Observable Conditions

Contact Info

Product

Resources

About