Towards Emotion-aided Multi-modal Dialogue Act Classification

Saha, Tulika; Patra, Aditya Prakash; Saha, Sriparna; Bhattacharyya, Pushpak

doi:10.18653/v1/2020.acl-main.402

Cited by 57 publications

(27 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dialogue acts can be predicted both by prosodic and lexical cues, with the latter being more effective [99,101]. Moreover, Saha et al [36] recently investigated the relationship between dialogue acts and text-based emotion recognition using two emotional datasets, one of them IEMOCAP, and found that jointly modelling both can improve emotion recognition performance. They released their dialogue act annotations 3 , thus opening a new avenue of exploration for the interaction between the underlying linguistic content and model performance.…”

Section: Dialogue Act Analysismentioning

confidence: 99%

Multistage linguistic conditioning of convolutional layers for speech emotion recognition

Triantafyllopoulos¹,

Reichel²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this contribution, we investigate the effectiveness of deep fusion of text and audio features for categorical and dimensional speech emotion recognition (SER). We propose a novel, multistage fusion method where the two information streams are integrated in several layers of a deep neural network (DNN), and contrast it with a single-stage one where the streams are merged in a single point. Both methods depend on extracting summary linguistic embeddings from a pre-trained BERT model, and conditioning one or more intermediate representations of a convolutional model operating on log-Mel spectrograms. Experiments on the widely used IEMOCAP and MSP-Podcast databases demonstrate that the two fusion methods clearly outperform a shallow (late) fusion baseline and their unimodal constituents, both in terms of quantitative performance and qualitative behaviour. Our accompanying analysis further reveals a hitherto unexplored role of the underlying dialogue acts on unimodal and bimodal SER, with different models showing a biased behaviour across different acts. Overall, our multistage fusion shows better quantitative performance, surpassing all alternatives on most of our evaluations. This illustrates the potential of multistage fusion in better assimilating text and audio information.

show abstract

Section: Dialogue Act Analysismentioning

confidence: 99%

Multistage linguistic conditioning of convolutional layers for speech emotion recognition

Triantafyllopoulos¹,

Reichel²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Emotion recognition is broadly studied with uni-modal (Yang et al 2018;Majumder et al 2019;Saha et al 2020;Jiao, Lyu, and King 2020;Huang et al 2021), bi-modal (Mittal et al 2020b;Liu et al 2020;Zhao et al 2020) and multi-modal (Mittal et al 2020a;Sun et al 2020;Zhang et al 2021a;Lv et al 2021). More effective multi-modal fusion translates to better performance.…”

Section: Related Workmentioning

confidence: 99%

Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition

Zhang¹,

Chen²,

Shen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), aiming to refine multi-modal representations and enhance discriminative capacity of each label. Specifically, we design an adversarial multi-modal refinement module to sufficiently explore the commonality among different modalities and strengthen the diversity of each modality. To further exploit label-modal dependence, we devise a BERT-like cross-modal encoder to gradually fuse private and common modality representations in a granularity descent way, as well as a label-guided decoder to adaptively generate a tailored representation for each label with the guidance of label semantics. In addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in both aligned and unaligned settings, which demonstrate the superiority of TAILOR over the state-of-the-arts. Code is available at https://github.com/kniter1/TAILOR.

show abstract

“…Thus, some researchers argue that chatbots should also have such abilities to blend information from different modalities. There are some recent works trying to build multimodal dialogue systems (Le et al, 2019;Chauhan et al, 2019;Saha et al, 2020;Singla et al, 2020;Young et al, 2020), but these systems are still far from mature.…”

Section: Conclusion and Trendsmentioning

confidence: 99%

Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey

Ni¹,

Young²,

Pandelea³

et al. 2021

Preprint

View full text Add to dashboard Cite

Dialogue systems are a popular Natural Language Processing (NLP) task as it is promising in real-life applications. It is also a complicated task since many NLP tasks deserving study are involved. As a result, a multitude of novel works on this task are carried out, and most of them are deep learning based due to the outstanding performance. In this survey, we mainly focus on the deep learning based dialogue systems. We comprehensively review state-of-the-art research outcomes in dialogue systems and analyze them from two angles: model type and system type. Specifically, from the angle of model type, we discuss the principles, characteristics, and applications of different models that are widely used in dialogue systems. This will help researchers acquaint these models and see how they are applied in state-of-the-art frameworks, which is rather helpful when designing a new dialogue system. From the angle of system type, we discuss task-oriented and open-domain dialogue systems as two streams of research, providing insight into the hot topics related. Furthermore, we comprehensively review the evaluation methods and datasets for dialogue systems to pave the way for future research. Finally, some possible research trends are identified based on the recent research outcomes. To the best of our knowledge, this survey is the most comprehensive and up-to-date one at present in the area of dialogue systems and dialogue-related tasks, extensively covering the popular frameworks, topics, and datasets 1 . * Equal contribution ‡ Corresponding author 1 The frameworks, topics, and datasets discussed are originated from the extensive literature review of state-of-the-art research. We have tried our best to cover all but may still omit some works. Readers are welcome to provide suggestions regarding the omissions and mistakes in this article. We also intend to update this article with time as and when new approaches or definitions are proposed and used by the community Preprint. Under review.

show abstract

Towards Emotion-aided Multi-modal Dialogue Act Classification

Cited by 57 publications

References 42 publications

Multistage linguistic conditioning of convolutional layers for speech emotion recognition

Multistage linguistic conditioning of convolutional layers for speech emotion recognition

Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition

Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey

Contact Info

Product

Resources

About