Multimodal Dialogue Management for Multiparty Interaction with Infants

Abstract

We present dialogue management routines for a system to engage in multiparty agent-infant interaction. The ultimate purpose of this research is to help infants learn a visual sign language by engaging them in naturalistic and socially contingent conversations during an early-life critical period for language development (ages 6 to 12 months) as initiated by an artificial agent. As a first step, we focus on creating and maintaining agent-infant engagement that elicits appropriate and socially contingent responses from the baby. Our system includes two agents, a physical robot and an animated virtual human. The system's multimodal perception includes an eye-tracker (measures attention) and a thermal infrared imaging camera (measures patterns of emotional arousal). A dialogue policy is presented that selects individual actions and planned multiparty sequences based on perceptual inputs about the baby's internal changing states of emotional engagement. The present version of the system was evaluated in interaction with 8 babies. All babies demonstrated spontaneous and sustained engagement with the agents for several minutes, with patterns of conversationally relevant and socially contingent behaviors. We further performed a detailed case-study analysis with annotation of all agent and baby behaviors. Results show that the baby's behaviors were generally relevant to agent conversations and contained direct evidence for socially contingent responses by the baby to specific linguistic samples produced by the avatar. This work demonstrates the potential for language learning from agents in very young babies and has especially broad implications regarding the use of artificial agents with babies who have minimal language exposure in early life.

Thermal Infrared Imaging-Based Affective Computing and Its Application to Facilitate Human Robot Interaction: A Review

“…This study was part of a larger project designed to develop a system called RAVE (Robot AVatar thermal Enhanced language learning tool). RAVE is aimed to be an augmentative learning tool that can provide linguistic input, in particular visual language inputs, to facilitate language learning during one widely recognized critical developmental period for language (ages 6-12 months [97]) [96,[98][99][100][101]. To this end, thermal IR imaging was used to determine the emotional arousal and attentional valence, providing new knowledge about when infants are most optimally "Ready to Learn", even before the onset of language production.…”

Can a Signing Virtual Human Engage a Baby's Attention?

“…Thus, this is well past the early critical period for learning phonological units, phonological segmentation, categorization and mapping, and sequencing distributions -all vital to optimal, healthy language learning and reading. As such, there is a pressing opportunity for AI technology that can provide signed language input in the critical period of 6-12 months [20,21,34].…”

“…The RAVE system includes two behavioral agents (a physical robot and a virtual human avatar on a screen) that can provide visual behaviors, as well as several sensor devices: an eye-tracker, thermal camera, and an interface for indicating communicative baby behaviors. Detailed description of the system's constituent components and dialogue algorithms are presented in [34], and [20], respectively. A preliminary evaluation of the system has been presented in [21].…”

“…One recently introduced AI system, called RAVE (Robot, AVatar, thermal Enhanced language learning tool), was designed specifically for babies within the age range of 6-12 months [20,21,34]. RAVE consists of two agents: a virtual human (provides language and socially contingent interactions) and an embodied robot (provides socially engaging physical cues to babies and directs babies' attention to the virtual human).…”

EyeFlow

“…While remote eye trackers allow estimating user gaze with high accuracy on computer screens, they are limited to their narrow ield-of-view and cannot be used with smaller surfaces (e.g., smartwatches). Eye tracking devices are used in experiments involving humans, even including infants [Franchak et al, 2011, Nasihati Gilani et al, 2018, that must not be harmed. Most pupil detection algorithms rely on head mounted eye cameras using an infrared light emitter attached next to the eye camera in order to seamlessly extract the pupil from the iris area in the imaging frame ( Figure 1).…”