Establishing Human Observer Criterion in Evaluating Artificial Social Intelligence Agents in a Search and Rescue Task

Abstract

Artificial social intelligence (ASI) agents have great potential to aid the success of individuals, human–human teams, and human–artificial intelligence teams. To develop helpful ASI agents, we created an urban search and rescue task environment in Minecraft to evaluate ASI agents’ ability to infer participants’ knowledge training conditions and predict participants’ next victim type to be rescued. We evaluated ASI agents’ capabilities in three ways: (a) comparison to ground truth—the actual knowledge training condition and participant actions; (b) comparison among different ASI agents; and (c) comparison to a human observer criterion, whose accuracy served as a reference point. The human observers and the ASI agents used video data and timestamped event messages from the testbed, respectively, to make inferences about the same participants and topic (knowledge training condition) and the same instances of participant actions (rescue of victims). Overall, ASI agents performed better than human observers in inferring knowledge training conditions and predicting actions. Refining the human criterion can guide the design and evaluation of ASI agents for complex task environments and team composition.

Factor Analysis of a Generalized Video Game Experience Measure

“…The data analyzed here was collected by Arizona State University as part of the Artificial Social Intelligence Supporting Teams project’s second experimental study (referred to as Study 2) and represents a portion of a larger dataset (available here: Huang et al, 2022). The study utilized multiple survey measures that capture various aspects of participants such as personality and social intelligence – of note for the present analyses were surveys related to demographic information as well as video game experience.…”

“…Participants. The current analyses use a subset of the entire ASIST dataset (a description and data files of the entire ASIST data collected is available here: Huang et al, 2022). The study collected 67 teams of three individuals, with three teams were removed due to errors in experiment administration, leaving 192 participants in the remaining sets.…”

“…Participants. The current analyses use a subset of the entire ASIST dataset (a description and data files of the entire ASIST data collected is available here: Huang et al, 2022b). Study 3 data consists of 336 participants.…”

Transdisciplinary Team Science: Transcending Disciplines to Understand Artificial Social Intelligence in Human-Agent Teaming

“…ASI acted as advisors, supporting teams as they worked to complete their missions by keeping tabs on their objectives and course of action, and sending text messages to intervene when the agent deemed necessary. A description of the testbed as an experimental platform can be found in Carrol et al (2021) and full description of the testbed environment can be found in Huang et al (2022).…”