A Multimodal Model for Predicting Conversational Feedbacks

Abstract

We propose in this paper a statistical model in the perspective of predicting listener's feedbacks in a conversation. The first contribution of the paper is a study of the prediction of all feedbacks, including those in overlap with the speaker with a good accuracy. Existing model are good at predicting feedbacks during a pause, but reach a very low success level for all feedbacks. We give in this paper a first step towards this complex problem. The second contribution is a model predicting precisely the type of the feedback (generic vs. specific) as well as other specific features (valence expectation) useful in particular for generating feedbacks in dialogue systems. This work relies on an original corpus.

Predicting Backchannel Signaling in Child-Caregiver Multimodal Conversations

“…Visual features. We included features that have been used in previous work on BC such as head movement (nodding and shaking) [2,5,21,28], gaze [17,21,38], eyebrow movements (raising and frowning) [21,22], and facial expressions (smiling and laughing) [2,21]. These features were annotated manually and were available with the dataset (for details, see [3]).…”

“…We selected a subset of the eGeMAPS that were used in several previous studies. We included pitch (variation) [5,21,22,27,28,40], Mel-Frequency Cepstral Coefficients (MFCC) [17,21,28,34], voice quality [17,21,28,34], energy [18,31,34] and pausal information [5,6,21,32]. To minimize identify-confounding [29] the features were centered and scaled for each participant.…”

“…First, we use Part-Of-Speech (POS) as they could indicate, e.g., whether an utterance has ended or whether the speaker is just making a pause. This feature is often associated with transitions between discourse units and may elicit communicative responses [5,6,38]. We extracted POS tags using SpaCy toolkit [20], resulting in 17 tags for our French dataset.…”

Didn't see that coming: a survey on non-verbal social human behavior forecasting

“…Most common multimodal settings combined landmarks, body/head pose, or visual cues with past utterance transcriptions (Chu et al, 2018;Hua et al, 2019;Ueno et al, 2020;), acoustic features (Türker et al, 2018Ahuja et al, 2019;Ueno et al, 2020;Goswami et al, 2020;Woo et al, 2021;Jain and Leekha, 2021;Murray et al, 2021;Ben-Youssef et al, 2021), speaker's metadata (Raman et al, 2021;, or with combinations of the previous modalities (Ishii et al, 2020;Huang et al, 2020;Blache et al, 2020;Ishii et al, 2021;Boudin et al, 2021). The most common way to exploit different modalities together consists in simply concatenating their embedded representations.…”

“…More recently, the collection, annotation and release of bigger datasets favored the appearance of data-driven automated multimodal methods for backchannel prediction. For example, Boudin et al (2021) used a logistic classifier that was trained on visual cues, prosodic and lexico-syntactic features in order to predict not only the backchannel opportunity but also their subtype associated (generic, positive, or expected). The choice of such a simple classifier was driven by the small dataset available.…”