A probabilistic multimodal approach for predicting listener backchannels

Abstract

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model or Conditional Random Fields) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.Keywords Listener backchannel feedback · Nonverbal behavior prediction · Sequential probabilistic model · Conditional random field · Head nod · Multimodal

Analyzing Perceived Empathy Based on Reaction Time in Behavioral Mimicry

“…Furthermore, the dynamics of action and reaction have been utilized for generating the reactions of a conversational avatar to a user. For example, Morency et al [25] proposed a probabilistic model for predicting the timing of listener's head gesture by analyzing the time lag of the listener's backchannel against speaker's utterance in human-human dyad conversations. However, none of them did not address how such interactions are perceived by observers.…”

When to Elicit Feedback in Dialogue: Towards a Model Based on the Information Needs of Speakers

“…Similarly, the second problem may be remedied by clustering feedback (places in the speaker's speech that are followed by feedback signals from multiple listeners are more likely to contain a cue). Nevertheless, the form-features in feedback elicitation cues have proven informative enough to enable automatic detection of feedback elicitation cues in audiovisual datastreams and have been successfully used to model the feedback behaviour of virtual agents [17,20].…”

Co-constructing Grounded Symbols—Feedback and Incremental Adaptation in Human–Agent Dialogue

“…So far, most research on feedback in dialogue agents has concentrated on the first of these two aspects, with particular emphasis placed on models of appropriate timing of backchannel feedback [26,20,18]. Recently, the increase in the capabilities of incremental natural language understanding, has directed attention to the question of what type of feedback should be provided [19,24,23].…”