Emo-MG Framework: LSTM-based Multi-modal Emotion Detection through Electroencephalography Signals and Micro Gestures

Fang, Le; Xing, Sark Pangrui; Ma, Zeyang; Zhang, Zhijie; Long, Yonghao; Lee, Kun Pyo; Wang, Stephen Jia

doi:10.1080/10447318.2023.2228983

Cited by 4 publications

(1 citation statement)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sarcasm detection, the task of automatically identifying sarcastic language, has received significant research attention due to its practical applications in sentiment analysis, opinion mining, and social media analysis. Compared to other forms of sentiment analysis, the complexity of sarcasm detection lies in the difficulty of identifying its appropriate contextual dependencies or distinguishing between literal meaning and the author's underlying intent.Some studies have explored the combination of cognitive and physiological features of humans, such as Electroencephalo-gram (EEG) signals [10,11] , facial data [12,13], expression habit [14] and micro-gestures [15]. Compared to physiological feature data, multimodal data from social networks is more easily obtainable and contains richer semantic information.…”

Section: Multi-modal Sarcasm Detectionmentioning

confidence: 99%

Granularity Based Inter and Intra-Modal Fusion Network for Sarcasm Detection

Shi,

Zhao,

Chen

2023

Preprint

View full text Add to dashboard Cite

Multi-modal sarcasm detection is a task that involves detecting and identifyingsarcasm using multiple modalities of information. The key aspect of this task lies in how to model intra and inter-modality incongruity. Existing multi-modal sarcasm detection methods often focus on incongruity between modalities while overlooking the potential of fully exploring the semantic information within each modality. We tackle the problem in this paper by designing the Granularity Based Inter and Intra-Modal Fusion Network (GIIFN). Our approach combines tradi-tional image algorithms with deep learning models to extract comprehensive andrich semantic information from images. By incorporating a pre-trained languagemodel, we leverage knowledge from large-scale textual data to enhance the anal-ysis of images. Furthermore, our feature interaction model can effectively fuse and interact features at different granularities, capturing fine details and contex-tual information in the images. Through extensive experimental validation, we demonstrate the outstanding performance of our approach in multimodal satiredetection tasks, surpassing existing methods and outperforming state-of-the-art results.

show abstract