Multi-cue Contingency Detection

Abstract

The ability to detect a human's contingent response is an essential skill for a social robot attempting to engage new interaction partners or maintain ongoing turn-taking interactions. Prior work on contingency detection focuses on single cues from isolated channels, such as changes in gaze, motion, or sound. We propose a framework that integrates multiple cues for detecting contingency from multimodal sensor data in human-robot interaction scenarios. We describe three levels of integration and discuss our method for performing sensor fusion at each of these levels. We perform a Wizard-of-Oz data collection experiment in a turn-taking scenario in which our humanoid robot plays the turn-taking imitation game "Simon says" with human partners. Using this data set, which includes motion and body pose cues from a depth and color image and audio cues from a microphone, we evaluate our contingency detection module with the proposed integration mechanisms and show gains in accuracy of our multi-cue approach over single-cue contingency detection. We show the importance of selecting the appropriate level of cue integration as well as the implications of varying the referent event parameter.ONR YIP N000140810842.

Simon plays Simon says: The timing of turn-taking in an imitation game

“…Turntaking is a highly multimodal process, and prior work gives much in-depth analysis of specific channels, such as gaze usage to designate speaker or listener roles [6] or speech strategies in spoken dialog systems [7]. Closely related is the problem of contingency or engagement detection, which requires implementing robot perception for awareness of the human's cue usage [8], [9], [10]. Turn-taking has also been demonstrated in situated agents [11], including management of multi-party conversation [12].…”

Getting interaction theory (IT) together

“…In Froese and Di Paolo's (2008) model, the agents do not realize their sensitivity to the other's responsiveness in terms of any internal ToM mechanism, or a 'social contingency detection module' (Lee et al 2012). Instead, their behavior is realized by means of collective properties of the interaction process, thereby demonstrating in principle that it is not always necessary to postulate specialized subpersonal cognitive mechanisms in order to explain the social capacities of an individual.…”

“…Most explanations of the infants' behavior have focused on individual cognitive capacities and subpersonal mechanisms, as is expected from the perspective of ToM approaches. For example, some research in social robotics has been geared toward the implementation of 'social contingency modules' (Lee, Chao, Bobick & Thomaz 2012). But is this methodological individualistic approach the only way in which to explain the empirical results?…”

Evaluating the Engagement with Social Robots

“…Lastly, humans may not re- spond at all, because of a complete inability of the robot to communicate with nonverbal-behaviors or gestures. The synchrony of human-robot movements and the contingent response to reciprocal cues are critical features for evaluating the quality of imitation and joint attention [48,30]. Figure 8 highlights the joint attention elicited by the robot towards a human: in blue the yaw head movement of the human, in red the robot's.…”