Multimodal Goal Recognition in Open-World Digital Games

Plan recognition is a key component of player modeling. Player plan recognition focuses on modeling how and when players select goals and formulate action sequences to achieve their goals during gameplay. By occasionally asking players to describe their plans, it is possible to devise robust plan recognition models that jointly reason about player goals and action sequences in coordination with player input. In this work, we present a player plan recognition framework that leverages data from player interactions with a planning support tool embedded in an educational game for middle school science education, CRYSTAL ISLAND. Players are prompted to use the planning tool to describe their goals and planned actions in CRYSTAL ISLAND. We use this data to devise data-driven player plan recognition models using multi-label multi-task learning. Specifically, we compare single-task and multi-task learning approaches for both goal prediction and action sequence prediction. Results indicate that multi-task learning yields significant benefits for action sequence prediction. Additionally, we find that incorporating automated detectors of plan completion in plan recognition models improves predictive performance in both tasks.

Section: Related Workmentioning

confidence: 99%

Section: Framework Inputmentioning

confidence: 99%

Robust Player Plan Recognition in Digital Games with Multi-Task Multi-Label Learning

Goslen

Carpenter

Rowe

et al. 2022

“…• Block #4 -Estimated Player Policy: To increase the reliability of an available rollout function, the manager might use an estimated player policy to estimate which action(s) the player might perform next, given their prior gameplay history. An example of this block can be found in a study by Min et al (2016), where they attempted to model how players would form and pursue new goals based on their prior experiences in the game. • Block #5 -Feature Vector: Given a trajectory of a player's prior (or potential future) experience in the game, it is common for managers to extract higher-level information that can aid in their reasoning process.…”

Section: Conceptual Frameworkmentioning

confidence: 99%

A Structured Analysis of Experience Management Techniques

Mori

Thue

Schiffel

2019

As a field of study, Experience Management spans a set of technology that is becoming increasingly relevant in applications that aim to improve the experiences of their users. Given the youth of the field, however, few attempts have been made to identify and discuss the common elements of systems that manage user experiences as they occur. In this paper, we consider a subset of existing AI experience managers in the context of a shared conceptual framework. We offer directly comparable summaries of the managers that we discuss, and we highlight how some of them use different technologies to perform similar tasks. While many experience managers remain undiscussed, we nonetheless demonstrate that distinct, well-defined components exist across a diverse set of managers, and that a modular method for building new managers might well be within reach.

“…Prior work has explored the use of traditional machine learning algorithms with handcrafted features to address this (Ha et al, 2011;Mott et al, 2006). Recent work using deep learning methods has eliminated the need for feature engineering and allowed for more generalizable methods (Min et al, 2014;Min et al, 2016b;Min et al, 2017a).…”

Section: Introductionmentioning

confidence: 99%

“…In addition to low-level gameplay actions, other modalities such as eye gaze data (Min et al, 2017b) have been investigated to improve the performance of goal recognition models by leveraging an expanded set of predictive features that may also indicate player goals. Another source of evidence that may support improved goal recognition in openworld games is students' natural language, written reflections during and after gameplay.…”

Section: Introductionmentioning

confidence: 99%

Enhancing Multimodal Goal Recognition in Open-World Games with Natural Language Player Reflections

Gupta

Carpenter

Min

et al. 2022

Open-world games promote engagement by offering players a high degree of autonomy to explore expansive game worlds. Player goal recognition has been widely explored for modeling player behavior in open-world games by dynamically recognizing players’ goals using observations of in-game actions and locations. In educational open-world games, in-game reflection tools can help students reflect on their learning and plan their strategies for future gameplay. Data generated from students’ written reflections can serve as a source of evidence for modeling player goals. We present a multimodal goal recognition approach that leverages players’ written reflections along with game trace log features to predict player goals during gameplay. Results show that both the highest predictive performance and best early prediction performance are achieved by deep learning-based, multimodal goal recognition models that utilize both written reflection and gameplay features as input. These models outperform unimodal deep learning models as well as a random forest baseline. Multimodal goal recognition using natural language reflection data has significant potential to enhance goal recognition model performance, as well as player modeling more generally, to support the creation of engaging and adaptive open-world digital games.