Recurrent Neural Networks for Emotion Recognition in Video

Kahou, Samira Ebrahimi; Michalski, Vincent; Konda, Kishore Reddy; Memisevic, Roland; Pal, Christopher

doi:10.1145/2818346.2830596

Cited by 271 publications

(70 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…AdaLBP (Zhao et al 2011) 73.54 Atlases (Guo, Zhao, and Pietikinen 2012) 75.52 ExpLet (Liu et al 2016) 76.65 Dis-ExpLet (Liu et al 2014) 79.00 Lomo (Sikka, Sharma, and Bartlett 2016) 82.10 DTAGN (Jung et al 2015) 81 Table 2 shows a comparison with previously recorded methods(including spatio-temporal and appearance based methods) (Elaiwat, Bennamoun, and Boussaid 2016;Kacem et al 2017;Hu et al 2017;Vielzeuf, Pateux, and Jurie 2017;Yao et al 2016;Ebrahimi Kahou et al 2015). On the other hand, no previously recorded methods are available on the KAIST Face MPMI datasets.…”

Section: Methodsmentioning

confidence: 99%

Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Baddar

2019

AAAI

View full text Add to dashboard Cite

Spatio-temporal feature encoding is essential for encoding the dynamics in video sequences. Recurrent neural networks, particularly long short-term memory (LSTM) units, have been popular as an efficient tool for encoding spatio-temporal features in sequences. In this work, we investigate the effect of mode variations on the encoded spatio-temporal features using LSTMs. We show that the LSTM retains information related to the mode variation in the sequence, which is irrelevant to the task at hand (e.g. classification facial expressions). Actually, the LSTM forget mechanism is not robust enough to mode variations and preserves information that could negatively affect the encoded spatio-temporal features. We propose the mode variational LSTM to encode spatio-temporal features robust to unseen modes of variation. The mode variational LSTM modifies the original LSTM structure by adding an additional cell state that focuses on encoding the mode variation in the input sequence. To efficiently regulate what features should be stored in the additional cell state, additional gating functionality is also introduced. The effectiveness of the proposed mode variational LSTM is verified using the facial expression recognition task. Comparative experiments on publicly available datasets verified that the proposed mode variational LSTM outperforms existing methods. Moreover, a new dynamic facial expression dataset with different modes of variation, including various modes like pose and illumination variations, was collected to comprehensively evaluate the proposed mode variational LSTM. Experimental results verified that the proposed mode variational LSTM encodes spatio-temporal features robust to unseen modes of variation.

show abstract

Section: Methodsmentioning

confidence: 99%

Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Baddar

2019

AAAI

View full text Add to dashboard Cite

show abstract

“…For a more detailed elaboration on emotion models in the context of music, we refer to [159,186]. e implicit acquisition of emotional states can be e ected, for instance, by analyzing user-generated text [50], speech [67], or facial expressions in video [56].…”

Section: Personalitymentioning

confidence: 99%

Current challenges and visions in music recommender systems research

Schedl

Zamani

Chen

et al. 2018

Int J Multimed Info Retr

260

155

View full text Add to dashboard Cite

Music recommender systems (MRS) have experienced a boom in recent years, thanks to the emergence and success of online streaming services, which nowadays make available almost all music in the world at the user's ngertip. While today's MRS considerably help users to nd interesting music in these huge catalogs, MRS research is still facing substantial challenges. In particular when it comes to build, incorporate, and evaluate recommendation strategies that integrate information beyond simple user-item interactions or content-based descriptors, but dig deep into the very essence of listener needs, preferences, and intentions, MRS research becomes a big endeavor and related publications quite sparse. e purpose of this trends and survey article is twofold. We rst identify and shed light on what we believe are the most pressing challenges MRS research is facing, from both academic and industry perspectives. We review the state of the art towards solving these challenges and discuss its limitations. Second, we detail possible future directions and visions we contemplate for the further evolution of the eld. e article should therefore serve two purposes: giving the interested reader an overview of current challenges in MRS research and providing guidance for young researchers by identifying interesting, yet under-researched, directions in the eld.

show abstract

“…Word embeddings are widely applied in sentiment analysis with neural network models [2,3,19]. However, these models often lack clear interpretations of word vectors [7].…”

Section: Related Workmentioning

confidence: 99%

Disentangling Latent Emotions of Word Embeddings on Complex Emotional Narratives

Jiang

2019

Natural Language Processing and Chinese Computing

View full text Add to dashboard Cite

Word embedding models such as GloVe are widely used in natural language processing (NLP) research to convert words into vectors. Here, we provide a preliminary guide to probe latent emotions in text through GloVe word vectors. First, we trained a neural network model to predict continuous emotion valence ratings by taking linguistic inputs from Stanford Emotional Narratives Dataset (SEND). After interpreting the weights in the model, we found that only a few dimensions of the word vectors contributed to expressing emotions in text, and words were clustered on the basis of their emotional polarities. Furthermore, we performed a linear transformation that projected high dimensional embedded vectors into an emotion space. Based on NRC Emotion Lexicon (EmoLex), we visualized the entanglement of emotions in the lexicon by using both projected and raw GloVe word vectors. We showed that, in the proposed emotion space, we were able to better disentangle emotions than using raw GloVe vectors alone. In addition, we found that the sum vectors of different pairs of emotion words successfully captured expressed human feelings in the EmoLex. For example, the sum of two embedded word vectors expressing Joy and Trust which express Love shared high similarity (similarity score .62) with the embedded vector expressing Optimism. On the contrary, this sum vector was dissimilar (similarity score -.19) with the the embedded vector expressing Remorse. In this paper, we argue that through the proposed emotion space, arithmetic of emotions is preserved in the word vectors. The affective representation uncovered in emotion vector space could shed some light on how to help machines to disentangle emotion expressed in word embeddings.

show abstract

Recurrent Neural Networks for Emotion Recognition in Video

Cited by 271 publications

References 18 publications

Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Current challenges and visions in music recommender systems research

Disentangling Latent Emotions of Word Embeddings on Complex Emotional Narratives

Contact Info

Product

Resources

About