Evaluation of Goal Recognition Systems on Unreliable Data and Uninspectable Agents

Abstract

Goal or intent recognition, where one agent recognizes the goals or intentions of another, can be a powerful tool for effective teamwork and improving interaction between agents. Such reasoning can be challenging to perform, however, because observations of an agent can be unreliable and, often, an agent does not have access to the reasoning processes and mental models of the other agent. Despite this difficulty, recent work has made great strides in addressing these challenges. In particular, two Artificial Intelligence (AI)-based approaches to goal recognition have recently been shown to perform well: goal recognition as planning, which reduces a goal recognition problem to the problem of plan generation; and Combinatory Categorical Grammars (CCGs), which treat goal recognition as a parsing problem. Additionally, new advances in cognitive science with respect to Theory of Mind reasoning have yielded an approach to goal recognition that leverages analogy in its decision making. However, there is still much unknown about the potential and limitations of these approaches, especially with respect to one another. Here, we present an extension of the analogical approach to a novel algorithm, Refinement via Analogy for Goal Reasoning (RAGeR). We compare RAGeR to two state-of-the-art approaches which use planning and CCGs for goal recognition, respectively, along two different axes: reliability of observations and inspectability of the other agent's mental model. Overall, we show that no approach dominates across all cases and discuss the relative strengths and weaknesses of these approaches. Scientists interested in goal recognition problems can use this knowledge as a guide to select the correct starting point for their specific domains and tasks.

Language Model-Based Player Goal Recognition in Open World Digital Games

“…1986;Charniak and Goldman 1993;Mirsky et al 2016;Treger and Kaminka 2022). Recently, Rabkina et al (2022) proposed a cognitive model-based goal recognition framework using the analogy of similar cases retrieved from a plan-library and evaluated it using MineCraft and Monroe. A key issue with these approaches is that as the problem space grows, it becomes increasingly difficult to generate and retrieve all possible plans.…”

Robust Player Plan Recognition in Digital Games with Multi-Task Multi-Label Learning

“…ToM is the ability to understand and predict intent, mental models and other cognitive characteristics, which is important when applied to player modeling (Shergadwala, Teng, and El-Nasr 2021). There has been a wide variety of methodologies for constructing such recognition problems that use ToM for plan and goal recognition tasks, such as recursive neural networks (Bisson, Larochelle, and Kabanza 2015), combinatory categorial grammars (Rabkina et al 2022), and hierarchical task networks (Rabkina et al 2021). While these methods have been shown to work well in digital games with pre-defined states, little work has been done on the potential of machine learning-based plan recognition in openworld digital games.…”