MEmoR: A Dataset for Multimodal Emotion Reasoning in Videos

Shen, Guangyao; Wang, Xin; Duan, Xuguang; Zhang, Lipeng; Zhu, Wenwu

doi:10.1145/3394171.3413909

Cited by 24 publications

(17 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Zero Padding (ZP): Padding the feature representations of the missing modality with zero is a widely used way to copy with incomplete modalities [30,31,32]. For this method, we consider two forms of φ to fuse features f and g: addition and concatenation.…”

Section: Baseline Methodsmentioning

confidence: 99%

Maximum Likelihood Estimation for Multimodal Learning with Missing Modality

Ma,

Xu,

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

Multimodal learning has achieved great successes in many scenarios. Compared with unimodal learning, it can effectively combine the information from different modalities to improve the performance of learning tasks. In reality, the multimodal data may have missing modalities due to various reasons, such as sensor failure and data transmission error. In previous works, the information of the modalitymissing data has not been well exploited. To address this problem, we propose an efficient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-missing data. Specifically, we design a likelihood function to characterize the conditional distribution of the modality-complete data and the modality-missing data, which is theoretically optimal. Moreover, we develop a generalized form of the softmax function to effectively implement maximum likelihood estimation in an end-to-end manner. Such training strategy guarantees the computability of our algorithm capably. Finally, we conduct a series of experiments on real-world multimodal datasets. Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality.Preprint. Under review.

show abstract

Section: Baseline Methodsmentioning

confidence: 99%

Maximum Likelihood Estimation for Multimodal Learning with Missing Modality

Ma,

Xu,

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…b) Emotional Conversation Datasets: Generally, the emotional perception ability of a dialogue model is defined as the task: emotion recognition in conversations (ERC) [40] or emotion reasoning (ER) [46]. Datasets, e.g., IEMOCAP [38], Mastodon [39], MELD [40], EMOTyDA [42], EDA [50], MEmoR [46] and M 3 ED [43], are usually used for the ERC or ER task. These datasets generally have small sizes, with fewer than 10K dialogues, making them unsuitable for conversation generation tasks.…”

Section: B Conversation Datasetsmentioning

confidence: 99%

“…PELD [48] is proposed for predicting emotion for response using BF personality traits and VAD vector, in which the personality traits are averaged with personality traits of FriendsPersona [47]. MEmoR [46], a recent multimodal emotion reasoning dataset used for the task of multimodal emotion reasoning, provides a multimodal conversation context, 14 fine-grained emotions and 3 types of personalities (16PF, BF and MBTI). MEmoR is mainly used for the task of multimodal emotion reasoning, in which the personalities are used for improving the performance of emotion reasoning.…”

Section: B Conversation Datasetsmentioning

confidence: 99%

“…In general, the emotion labels of conversation datasets are considered from among Ekman's six basic emotions (joy, sad, feared, angry, surprise, and disgusted) [54]. However, the latest studies, e.g., 32 emotion labels in EmpatheticDialogues [41] and 14 emotion labels in MEmoR [55], show that more fine-grained emotion annotation can contribute to research on emotional reasoning and empathetic conversation. Considering the diversity of emotional tags and the similarity of different tags, we selected 13 emotion labels referring to EmpatheticDialogues [41] and 19 DA labels referring to the SWBD-DAMSL tag-set [56] based on the characteristics of Chinese open-domain conversation.…”

Section: Cped Datasetmentioning

confidence: 99%

See 1 more Smart Citation

CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI

Chen¹,

Fan²,

Xing³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human language expression is based on the subjective construal of the situation instead of the objective truth conditions, which means that speakers' personalities and emotions after cognitive processing have an important influence on conversation. However, most existing datasets for conversational AI ignore human personalities and emotions, or only consider part of them. It's difficult for dialogue systems to understand speakers' personalities and emotions although large-scale pretraining language models have been widely used. In order to consider both personalities and emotions in the process of conversation generation, we propose CPED, a large-scale Chinese personalized and emotional dialogue dataset, which consists of multi-source knowledge related to empathy and personal characteristic. These knowledge covers gender, Big Five personality traits, 13 emotions, 19 dialogue acts and 10 scenes. CPED contains more than 12K dialogues of 392 speakers from 40 TV shows. We release the textual dataset with audio features and video features according to the copyright claims, privacy issues, terms of service of video platforms. We provide detailed description of the CPED construction process and introduce three tasks for conversational AI, including personality recognition, emotion recognition in conversations as well as personalized and emotional conversation generation. Finally, we provide baseline systems for these tasks and consider the function of speakers' personalities and emotions on conversation. Our motivation is to propose a dataset to be widely adopted by the NLP community as a new open benchmark for conversational AI research. The full dataset is available 1 .

show abstract

“…DialoGPT [2], CDialGPT [7]) can only learn personalized or emotional expressions through the dialogue context (single-modal or multi-modal) provided by the corpus. b) Emotional Conversation Datasets: Generally, the emotional perception ability of a dialogue model is defined as the task: emotion recognition in conversations (ERC) [40] or emotion reasoning (ER) [46]. Datasets, e.g., IEMOCAP [38], Mastodon [39], MELD [40], EMOTyDA [42], EDA [50], MEmoR [46] and M 3 ED [43], are usually used for the ERC or ER task.…”

Section: B Conversation Datasetsmentioning

confidence: 99%

CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI

Chen¹,

Fan²,

Xing³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p>Human language expression is based on the subjective construal of the situation instead of the objective truth conditions, which means that speakers' personalities and emotions after cognitive processing have an important influence on conversation. However, most existing datasets for conversational AI ignore human personalities and emotions, or only consider part of them. It's difficult for dialogue systems to understand speakers' personalities and emotions although large-scale pre-training language models have been widely used. In order to consider both personalities and emotions in the process of conversation generation, we propose CPED, a large-scale Chinese personalized and emotional dialogue dataset, which consists of multi-source knowledge related to empathy and personal characteristic. These knowledge covers gender, Big Five personality traits, 13 emotions, 19 dialogue acts and 10 scenes. CPED contains more than 12K dialogues of 392 speakers from 40 TV shows. We release the textual dataset with audio features and video features according to the copyright claims, privacy issues, terms of service of video platforms. We provide detailed description of the CPED construction process and introduce three tasks for conversational AI, including personality recognition, emotion recognition in conversations as well as personalized and emotional conversation generation. Finally, we provide baseline systems for these tasks and consider the function of speakers' personalities and emotions on conversation.</p> <p>Our motivation is to propose a dataset to be widely adopted by the NLP community as a new open benchmark for conversational AI research. The full dataset is available at https://github.com/scutcyr/CPED.</p>

show abstract

MEmoR: A Dataset for Multimodal Emotion Reasoning in Videos

Cited by 24 publications

References 31 publications

Maximum Likelihood Estimation for Multimodal Learning with Missing Modality

Maximum Likelihood Estimation for Multimodal Learning with Missing Modality

CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI

CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI

Contact Info

Product

Resources

About