Emotion-Aware Multimodal Pre-training for Image-Grounded Emotional Response Generation

Abstract

Face-to-face communication leads to better interactions between speakers than text-to-text conversations since the speakers can capture both textual and visual signals. Image-grounded emotional response generation (IgERG) tasks requires chatbots to generate a response with the understanding of both textual contexts and speakers' emotions in visual signals. Pre-training models enhance many NLP and CV tasks and image-text pre-training also helps multimodal tasks. However, existing image-text pre-training methods typically pre-train on images by recognizing or modeling objects, but ignore the emotions expressed in the images. In this paper, we propose several pre-training tasks in a unified framework that not only captures emotions from images but also learns to incorporate the emotion into text generation. The pre-training involves single-modal learning to strengthen the ability to understand images and generate texts. It also involves cross-modal learning to enhance interactions between images and texts. The experiments verify our method in appropriateness, informativeness, and emotion consistency.

Beyond Words: An Intelligent Human-Machine Dialogue System with Multimodal Generation and Emotional Comprehension

“…In their work, emotions were classifed into two broad categories, namely, positive and negative, to facilitate a simplifed emotional understanding. In a distinct study, Tian et al [10] put forth a multitask learning framework in which tasks such as image sentiment sequential labeling, image sentiment classifcation, and text generation were learned simultaneously. Tis was accomplished using a pretrained model specifcally designed to generate textual content that efectively captures the user's emotions.…”

“…In the realm of academia, researchers have extensively investigated dialogue models, such as those presented in Shuster et al [4,5], and have proposed emotion-enhanced models, as discussed in Wei et al [6] and Li et al [7]. Specifcally, to address the limitations of single text generation models, multimodal dialogue models capable of processing both textual and video information have been proposed, including the works of Fung et al [8], Huber et al [9], and Tian et al [10]. More importantly, Shen et al [11] designed ViDA-MAN, a digital human agent for multimodal interaction, which provides real-time audiovisual responses to users through voice queries.…”