A multimodal mixture-of-experts model for dynamic emotion prediction in movies

Goyal, Ankit; Kumar, Naveen; Guha, Tanaya; Narayanan, Shrikanth

doi:10.1109/icassp.2016.7472192

Cited by 24 publications

(32 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This ambiguity might prevent the model parameters for the couples in the "gray-zone" from being adequately learned. A way to address this limitation would be to perform soft clustering, according to which each couple is assigned to a probability of belonging to a given cluster, implemented through mixture of experts methodologies [27,62]. In this way, the parameters of each cluster will be learned using all samples, each weighted with an importance proportional to the strength of its belonging to a given cluster, potentially yielding more robust representations.…”

Section: Discussionmentioning

confidence: 99%

Sub-Population Specific Models of Couples’ Conflict

Gupta

Gujral

Chaspari

et al. 2020

ACM Trans. Internet Technol.

View full text Add to dashboard Cite

Interpersonal conflict between couples is a significant source of stress with long-lasting effects on partners’ physical and psychological health. Motivated by findings in psychological science, we study how couples with distinct relationship functioning characteristics experience conflict in real life. We propose sub-population specific machine learning models using hierarchical and adaptive learning frameworks to automatically detect interpersonal conflict through the ambulatory monitoring of couples’ physiological signals, audio samples, and linguistic indices. Results indicate that the proposed models outperform a general model learned for the entire population and separate models independently trained on each sub-population, providing a foundation toward personalized health applications.

show abstract

Section: Discussionmentioning

confidence: 99%

Sub-Population Specific Models of Couples’ Conflict

Gupta

Gujral

Chaspari

et al. 2020

ACM Trans. Internet Technol.

View full text Add to dashboard Cite

show abstract

“…When signals propagate through the network, a conditional layer learns which expert networks to activate, and so that the various combinations of experts are flexible under different circumstances. It was proven to outperform popular fusion strategies in dynamic emotion prediction using visual-audio [49], [50]. For developing DMoE model, we apply fully-connected layers for both expert and gating networks.…”

Section: Multimodal Fusion Modulesmentioning

confidence: 99%

Deep Graph Fusion Based Multimodal Evoked Expressions From Large-Scale Videos

Yang

Kim

et al. 2021

IEEE Access

View full text Add to dashboard Cite

In the case of emotion recognition for wild input signals with large variances, multiple sources of noise can challenge the machine's ability to learn approximate ground truth. There are copious studies on recognizing characters' affective expressions directly through face, speech, and text. However, there are few researches on the prediction of a characters' emotions from their watched contents. Therefore, in this paper, we propose a hybrid fusion model, called deep graph fusion, to leverage the combination of visual-audio representations for predicting viewers' evoked expressions from videos. The proposed system includes four steps. First, we extract features for visual and audio modalities for each 30-second segment using CNN-based pre-trained models to learn their salient representations. Then, we reconstruct these features into graph outlines, and conduct node embedding using graph convolutional networks. In the third steps, we propose various fusion modules to combine the graph representations from visual and audio branches. Finally, the fused features are applied to Sigmoid activation in order to estimate the evoked scores for all emotional classes. Moreover, in order to enhance the overall performance, we propose a semantic embedding loss to learn the semantic meaning of textual emotions. We evaluate the proposed method using the Evoked Expression from Videos (EEV) database on both the validation and test sets. The experimental results demonstrate that the proposed algorithm outperforms all baseline models.

show abstract

“…Over the past three years or so, higher level state prediction has also been investigated with various types of data collected from music [164][165][166], video [167,168] and EEG signals [169,170]. Despite the increasing interest in new modalities such as the EEG, most work in this area has concerned predictive medical applications [128,171], although a study on improving driving safety by predicting emotion from EGG signals can be found [51].…”

Section: Prediction Of Higher Level Individual Characteristicsmentioning

confidence: 99%

Predictive biometrics: a review and analysis of predicting personal characteristics from biometric data

Fairhurst

Abreu

2017

IET biom.

View full text Add to dashboard Cite

Interest in the exploitation of soft biometrics information has continued to develop over the last decade or so. In comparison with traditional biometrics, which focuses principally on person identification, the idea of soft biometrics processing is to study the utilisation of more general information regarding a system user, which is not necessarily unique. There are increasing indications that this type of data will have great value in providing complementary information for user authentication. However, the authors have also seen a growing interest in broadening the predictive capabilities of biometric data, encompassing both easily definable characteristics such as subject age and, most recently, 'higher level' characteristics such as emotional or mental states. This study will present a selective review of the predictive capabilities, in the widest sense, of biometric data processing, providing an analysis of the key issues still adequately to be addressed if this concept of predictive biometrics is to be fully exploited in the future.

show abstract

A multimodal mixture-of-experts model for dynamic emotion prediction in movies

Abstract: Please refer to published version for the most recent bibliographic citation information. If a published version is known of, the repository item page linked to above, will contain details on accessing it.

Cited by 24 publications

References 16 publications

Sub-Population Specific Models of Couples’ Conflict

Sub-Population Specific Models of Couples’ Conflict

Deep Graph Fusion Based Multimodal Evoked Expressions From Large-Scale Videos

Predictive biometrics: a review and analysis of predicting personal characteristics from biometric data

Contact Info

Product

Resources

About