Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents

Abstract

As it becomes more common for humans to work alongside artificial agents on everyday tasks, it is increasingly important to design artificial agents that can understand and interact with their human counterparts naturally. We posit that an effective way to do this is to harness nonverbal cues used in humanhuman interaction. We, therefore, leverage knowledge from existing work on gazebased intention recognition, where the awareness of gaze can provide insights into the future actions of an observed human subject. In this paper, we design and evaluate the use of a proactive intention-aware gaze-enabled artificial agent that assists a human player engaged in an online strategy game. The agent assists by recognising and communicating the intentions of a human opponent in real-time, potentially improving situation awareness. Our first study identifies the language requirements for the artificial agent to communicate the opponent's intentions to the assisted player, using an inverted Wizard of Oz method approach. Our second study compares the experience of playing an online strategy game with and without the assistance of the agent. Specifically, we conducted a within-subjects study with 30 participants to compare their experience of playing with (1) detailed AI predictions, (2) abstract AI predictions, and (3) no AI predictions but with a live visualisation of their opponent's gaze. Our results show that the agent can facilitate awareness of another user's intentions without adding visual distraction to the interface; however, the cognitive workload was similar across all three conditions, suggesting that the manner in which the agent communicates its predictions requires further exploration. Overall, our work contributes to the understanding of how to support human-agent teams in a dynamic collaboration scenario. We provide a positive account of humans interacting with an intention-aware artificial agent afforded by gaze input, which presents immediate opportunities for improving interactions between humans and agents and demonstrates the usefulness of human elicitation for producing communication protocols for effective interaction.

G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios

“…Reasoning. Gaze has been applied across various tasks related to cognitive reasoning, including collaborative robotics [8,21,38], interactive technology inputs [51,53,69], reading assistant [4,23], cognitive load measurement [50], and intent classification [24,31,45,72]. Eye movements is a reliable indicator of user intent -"eyes rarely visit objects that are irrelevant to the action" [35,62], which suggests a strong correlation between eye movements and inquiry intent in daily scenarios.…”

Designing Visual Markers for Continuous Artificial Intelligence Support

“…This results in a continuous interaction between (sustained) user input (e.g., moving a camera feed) and AI suggestions. Other recent examples from the HCI literature include the use of gaze-enabled intention recognition [38] and collaborative music improvisation [34] -showcasing how both implicit and explicit user input can drive continuous AI support systems. We illustrate these different interaction paradigms in Figure 8.…”