Generative Model of Agent’s Behaviors in Human-Agent Interaction

Dermouche, Soumia; Pélachaud, Catherine

doi:10.1145/3340555.3353758

Cited by 15 publications

(27 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although this study focused on facial expressions, other nonverbal behaviors are also important. As previous studies proposed [8,12], adding head pose prediction would be one possibility to improve the proposed method. In addition, predicting verbal feedback is also an important future direction.…”

Section: Discussionmentioning

confidence: 99%

“…In a study on adaptation in video-mediated communication, Feng et al [12] analyzed Skype chat videos and used LSTM to predict the next facial expression from previous ones for both participants. Dermouche et al [8] also used LSTM to predict Images of the other participants Self-image 遠隔議論の実験の様子(ＣＲ) Figure 2: Two displays were set in front of each participant; one large and one small. The larger display showed images of the other participants.…”

Section: Feedback Response Generationmentioning

confidence: 99%

See 1 more Smart Citation

Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication

Ueno

Nakano

Zeng

et al. 2020

Proceedings of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Providing feedback to a speaker is an essential communication signal for maintaining a conversation. In specific feedback, which indicates the listener's reaction to the speaker's utterances, the facial expression is an effective modality for conveying the listener's reactions. Moreover, not only the type of facial expressions, but also the degree of intensity of the expressions, may influence the meaning of the specific feedback. In this study, we propose a multimodal deep neural network model that predicts the intensity of facial expressions co-occurring with feedback responses. We focus on multiparty video-mediated communication. In video-mediated communication, close-up frontal face images of each participant are continuously presented on the display; the attention of the participants is more likely to be drawn to the facial expressions. We assume that in such communication, the importance of facial expression in the listeners' feedback responses increases. We collected 33 video-mediated conversations by groups of three people and obtained audio and speech data for each participant. Using the corpus collected as a dataset, we created a deep neural network model that predicts the intensity of 17 types of action units (AUs) co-occurring with the feedback responses. The proposed method employed GRU-based model with attention mechanism for audio, visual, and language modalities. A decoder was trained to produce the intensity values for the 17 AUs frame by frame. In the experiment, unimodal and multimodal models were compared in terms of their performance in predicting salient AUs that characterize facial expression in feedback responses. The results suggest that wellperforming models differ depending on the AU categories; audio information was useful for predicting AUs that express happiness,

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Feedback Response Generationmentioning

confidence: 99%

Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication

Ueno

Nakano

Zeng

et al. 2020

Proceedings of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

show abstract

“…Ekman & Friesen's Facial Action Coding System (FACS) [16] was developed for subject-independent coding of facial expressions for psychology research. It has also been widely used in graphics and machine-learning applications [11,14,24], but while FACS is well suited for coding, e.g., emotional expressions, it is less ideal for speech animation. There is also no canonical way of automatically encoding and decoding between video and FACS.…”

Section: Related Work 21 Representing Facial Communicative Signalsmentioning

confidence: 99%

“…They also lead to subjectspecific data and cannot easily be used in generation. MPEG-4 Face Animation Parameters (FAP) are closely related to FACS but were designed to cope with both analysis and synthesis and are, for example, used as output parameters in [14]. There is however a lack of reliable tools for reconstruction/synthesis.…”

Section: Related Work 21 Representing Facial Communicative Signalsmentioning

confidence: 99%

“…Furthermore, our system also relies on FLAME parameters for parametrization of the facial features as opposed to facial landmarks, granting several benefits; most importantly, the output parameters can directly generate a high-quality 3D face with corresponding gestures while simultaneously providing independent control over lip-sync and facial shape. Dermouche et al [14] presented a system similar to Feng et al but added the conversational state as additional conditional information, and also created a system usable in real time. They encoded the input using LSTMs while outputting FAPs.…”

Section: Interlocutor-aware Gesture Generationmentioning

confidence: 99%

See 1 more Smart Citation

Let's Face It

Jonell

Kucherenko

Henter

et al. 2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

View full text Add to dashboard Cite

To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal behavior for the agent, for example facial gestures, here defined as facial expressions and head movements. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. Those that do, typically use deterministic methods that risk producing repetitive and non-vivid motions. In this paper, we introduce a probabilistic method to synthesize interlocutor-aware facial gestures ś represented by highly expressive FLAME parameters ś in dyadic conversations. Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) a subjective evaluation assessing the use and relative importance of the different modalities in the synthesized output. The results show that the model successfully leverages the input from the interlocutor to generate more appropriate behavior. Videos, data, and code are available at: https://jonepatr.github.io/lets_face_it/.

show abstract

Immersive machine learning for social attitude detection in virtual reality narrative games

2022

View full text Add to dashboard Cite

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g. head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes.

show abstract

Generative Model of Agent’s Behaviors in Human-Agent Interaction

Cited by 15 publications

References 21 publications

Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication

Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication

Let's Face It

Immersive machine learning for social attitude detection in virtual reality narrative games

Contact Info

Product

Resources

About