Exploration meets exploitation: Multitask learning for emotion recognition based on discrete and dimensional models

Tu, Geng; Wen, Jintao; Líu, Hao; Chen, Sentao; Zheng, Lin; Jiang, Dazhi

doi:10.1016/j.knosys.2021.107598

Cited by 32 publications

(9 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applications such as social networks and news websites have generated abundant multimodal data. As a result, people have conducted extensive multimodal investigations; for example, sentiment analysis [23][24][25][26][27], image and text retrieval [28], reason extraction [29,30], and sarcasm detection [31]. Unlike text-modal-based sarcasm detection, multimodal sarcasm detection aims to identify sarcastic expressions implied in multimodal data.…”

Section: Multimodal Sarcasm Detectionmentioning

confidence: 99%

A Multi-View Interactive Approach for Multimodal Sarcasm Detection in Social Internet of Things with Knowledge Enhancement

Liu,

Yang,

2024

Applied Sciences

Self Cite

View full text Add to dashboard Cite

Multimodal sarcasm detection is a developing research field in social Internet of Things, which is the foundation of artificial intelligence and human psychology research. Sarcastic comments issued on social media often imply people’s real attitudes toward the events they are commenting on, reflecting their current emotional and psychological state. Additionally, the limited memory of Internet of Things mobile devices has posed challenges in deploying sarcastic detection models. An abundance of parameters also leads to an increase in the model’s inference time. Social networking platforms such as Twitter and WeChat have generated a large amount of multimodal data. Compared to unimodal data, multimodal data can provide more comprehensive information. Therefore, when studying sarcasm detection on social Internet of Things, it is necessary to simultaneously consider the inter-modal interaction and the number of model parameters. In this paper, we propose a lightweight multimodal interaction model with knowledge enhancement based on deep learning. By integrating visual commonsense knowledge into the sarcasm detection model, we can enrich the semantic information of image and text modal representation. Additionally, we develop a multi-view interaction method to facilitate the interaction between modalities from different modal perspectives. The experimental results indicate that the model proposed in this paper outperforms the unimodal baselines. Compared to multimodal baselines, it also has similar performance with a small number of parameters.

show abstract

Section: Multimodal Sarcasm Detectionmentioning

confidence: 99%

A Multi-View Interactive Approach for Multimodal Sarcasm Detection in Social Internet of Things with Knowledge Enhancement

Liu,

Yang,

2024

Applied Sciences

Self Cite

View full text Add to dashboard Cite

show abstract

“…Among the existing models, the circumplex model is most commonly used in affective computing. This approach easily estimates the similarities and differences between emotions because of the continuity of numerical vectors [12]. In the dimensional approach, emotions are conceptualized within two dimensions, namely arousal, and valence.…”

Section: Introductionmentioning

confidence: 99%

Comparison of Electrodermal Activity Signal Decomposition Techniques for Emotion Recognition

Veeranki,

Ganapathy,

Swaminathan

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Emotions play an essential role in human life as they are linked to well-being and markers of various diseases. Physiological signals can be used to assess emotions objectively and continuously. Electrodermal activity (EDA) is particularly interesting to assess emotions due to its relationship with the sympathetic nervous system. EDA signals are composed of tonic and phasic components that react differently to emotions, and various methods are available to obtain these components. However, the most accurate and effective method used for emotion analysis based on the phasic component of EDA has not been reported so far. This study presents the comparison of various EDA decomposition methods used for emotion detection based on levels of affective dimensions (arousal and valence) levels (low vs. high). In this study, EDA was decomposed using six methods, namely convex optimization-based EDA (cvxEDA), Time-Varying Sympathetic Activity (TVSymp), continuous decomposition analysis (CDA), dynamic causal modeling (DCM), BayesianEDA, and sparse deconvolution approach (Sparse). To test the most usable decomposition method for objective assessment of emotions, EDA signals from the database for emotion analysis using physiological signals (DEAP) were obtained. Statistical, morphological, Hjorth, and non-linear Entropy features were extracted from the phasic component obtained from each decomposition method and fed to the Random Forest and support vector machine classifiers for detection of arousal and valence affective dimension. TVSymp yielded the highest F1 score of 72.79% and 73.49% for classifying Arousal and Valence, respectively.

show abstract

“…Despite the progress, text alone cannot provide sufficient cues for deeper feelings compared to multimodal percep- tion (Hazarika et al 2018). Existing multimodal ERC methods mainly focus on aggregation-based fusion by concatenation (Tu et al 2022b), tensor product (Mai, Hu, and Xing 2019;Liu et al 2018), attention network (Rahman et al 2020;Wang et al 2019) or heterogeneous graph (Yang et al 2021;Hu et al 2022), etc. For instance, Hazarika et al (2018) proposed a conversational memory network to align features from multiple views.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Graph Learning for Multimodal Conversational Emotion Detection

Tu,

Xie,

Liang

et al. 2024

AAAI

View full text Add to dashboard Cite

Multimodal Emotion Recognition in Conversations (ERC) aims to identify the emotions conveyed by each utterance in a conversational video. Current efforts encounter challenges in balancing intra- and inter-speaker context dependencies when tackling intra-modal interactions. This balance is vital as it encompasses modeling self-dependency (emotional inertia) where speakers' own emotions affect them and modeling interpersonal dependencies (empathy) where counterparts' emotions influence a speaker. Furthermore, challenges arise in addressing cross-modal interactions that involve content with conflicting emotions across different modalities. To address this issue, we introduce an adaptive interactive graph network (IGN) called AdaIGN that employs the Gumbel Softmax trick to adaptively select nodes and edges, enhancing intra- and cross-modal interactions. Unlike undirected graphs, we use a directed IGN to prevent future utterances from impacting the current one. Next, we propose Node- and Edge-level Selection Policies (NESP) to guide node and edge selection, along with a Graph-Level Selection Policy (GSP) to integrate the utterance representation from original IGN and NESP-enhanced IGN. Moreover, we design a task-specific loss function that prioritizes text modality and intra-speaker context selection. To reduce computational complexity, we use pre-defined pseudo labels through self-supervised methods to mask unnecessary utterance nodes for selection. Experimental results show that AdaIGN outperforms state-of-the-art methods on two popular datasets. Our code will be available at https://github.com/TuGengs/AdaIGN.

show abstract

Exploration meets exploitation: Multitask learning for emotion recognition based on discrete and dimensional models

Cited by 32 publications

References 43 publications

A Multi-View Interactive Approach for Multimodal Sarcasm Detection in Social Internet of Things with Knowledge Enhancement

A Multi-View Interactive Approach for Multimodal Sarcasm Detection in Social Internet of Things with Knowledge Enhancement

Comparison of Electrodermal Activity Signal Decomposition Techniques for Emotion Recognition

Adaptive Graph Learning for Multimodal Conversational Emotion Detection

Contact Info

Product

Resources

About