Learning-based modeling of multimodal behaviors for humanlike robots

Abstract

In order to communicate with their users in a natural and effective manner, humanlike robots must seamlessly integrate behaviors across multiple modalities, including speech, gaze, and gestures. While researchers and designers have successfully drawn on studies of human interactions to build models of humanlike behavior and to achieve such integration in robot behavior, the development of such models involves a laborious process of inspecting data to identify patterns within each modality or across modalities of behavior and to represent these patterns as "rules" or heuristics that can be used to control the behaviors of a robot, but provides little support for validation, extensibility, and learning. In this paper, we explore how a learning-based approach to modeling multimodal behaviors might address these limitations. We demonstrate the use of a dynamic Bayesian network (DBN) for modeling how humans coordinate speech, gaze, and gesture behaviors in narration and for achieving such coordination with robots. The evaluation of this approach in a human-robot interaction study shows that this learning-based approach is comparable to conventional modeling approaches in enabling effective robot behaviors while reducing the effort involved in identifying behavioral patterns and providing a probabilistic representation of the dynamics of human behavior. We discuss the implications of this approach for designing natural, effective multimodal robot behaviors.

Interactive Drama With Robots for Teaching Non-Technical Subjects

“…Eye contact and head gaze give the impression that the robot is listening and paying attention to what people say or do (Andrist, Tan, Gleicher, & Mutlu, 2014;Hoffman et al, 2008;Huang & Mutlu, 2014;Srinivasan, Murphy, & Bethel, 2015).  Coherence between internal state and non-verbal expressions: The robot actor should define what gesture, body pose or physical action best describes the internal state and its context (Hoffman, 2011;Simmons et al, 2011).…”

Learning multimodal behavioral models for face-to-face social interaction

“…Quantitative evaluation showed that the LDCRF models achieved the best performance, underlying the importance of learning the dynamics between different gesture classes and the hidden internal orchestration of the gestures. Huang et al [28] explored how a learning-based approach meant to model multimodal behaviors might address the limitations of heuristic-based models. They used DBNs to model the coordination of speech, gaze, and gesture behaviors in narration.…”

Efficiency of speech and iconic gesture integration for robotic and human communicators - a direct comparison

“…Huang and Mutlu found that participants' recall of items in a factual talk presented by a robot was reliably improved if the robot used deictic gestures, while other types of gesture had little impact [21]. Bremner et al found that parts of a monologue accompanied by (metaphoric and beat) gestures were not recalled any better than those without, though higher certainty in the information recalled by the gestures was observed [22].…”