Attending to Emotional Narratives

Wu, Zhengxuan; Zhi-Xuan, Tan; Zaki, Jamil; Ong, Desmond C.

doi:10.1109/acii.2019.8925497

Cited by 14 publications

(14 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In its current form, the SENDv1 has proven to be a useful data set for training emotion recognition models; however, there is still room for future work on the modeling, in order to best extract and integrate the rich information from multiple modalities. For example, all our best-performing models used only one or two modalities, and future research could examine how to better integrate multimodal information to improve performance: We found in a recent investigation [49] that state-of-the-art models with simple concatenation fusion do poorly on multimodal inputs on the SENDv1, and required more sophisticated fusion methods to better integrate multiple modalities.…”

Section: Modelingmentioning

confidence: 98%

“…This subsequently led to a surge of interest in applying LSTMs, especially to time-series emotion recognition on the AVEC 2015 [42], [43], AVEC 2017 [44], [45], AVEC 2018 [46], and OMG-Empathy 2019 [47] challenges. Other noteworthy examples are [48], who investigated bidirectional LSTMs (where there is another recurrence that goes backwards in time), [49] who combined neural attention mechanisms with LSTMs, and [50] who built an LSTM with electroencephalography (EEG) input. These papers have collectively found that RNNs/LSTMs are a powerful model for time-series emotion recognition, whether they rely on extracted low-level features, or combined with features extracted using CNNs.…”

Section: Discriminative Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Ong

Zhi-Xuan

et al. 2021

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. We demonstrate several baseline and state-of-the-art modeling approaches on the SEND, including a Long Short-Term Memory model and a multimodal Variational Recurrent Neural Network, which perform comparably to the human-benchmark. We end by discussing the implications for future research in time-series affective computing.recognition. Specifically, we define time-series modeling as taking in temporally continuous input data and producing temporally continuous output, with an explicit consideration of how information is propagated over time. For instance, in order to engage in such inference, a social robot in conversation with its user would have to take in a continuous stream of sensor data, process them, and reason about their user's emotions over time, perhaps after every second or after every sentence, as well as across many sentences in the conversation and across multiple conversations [11].Despite the progress that has been made in time-series emotion recognition in the past decade, the field is still far from affective robots that can understand human emotions in daily life. What is needed to achieve this ambitious goal? We suggest that the biggest barriers to overcome are due to (1) the inherent difficulty of building computational time-series models, and (2) the difficulty of collecting highquality datasets. To address this first gap, we conduct a review covering different machine-learning-based approaches to time-series modeling (Section 2). We begin by discussing the most common time-series techniques in affective computing: deep neural network models, part of a broader class of discriminative models. We also cover generative time-series approaches, which are comparatively less popular within affective computing, but offer interesting modeling capabilities and hold exciting potential for emotion understanding.We turn next to discuss the second gap: Researchers need high-quality time-series datasets on which to train models. These are expensive to construct, in terms of both the production of stimuli and the collection of timeseries annotations of emotion and affective labeling [12]. There are several existing time-series datasets that have been used by the ...

show abstract

Section: Modelingmentioning

confidence: 98%

Section: Discriminative Modelsmentioning

confidence: 99%

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Ong

Zhi-Xuan

et al. 2021

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Word embeddings are widely applied in sentiment analysis with neural network models [2,3,19]. However, these models often lack clear interpretations of word vectors [7].…”

Section: Related Workmentioning

confidence: 99%

“…In this paper, we used Stanford Emotional Narratives Dataset (SEND) as our dataset. SEND is comprised of transcripts of video recordings in which participants shared emotional stories, and it has been well explored in computational models of emotion [12,19]. In each transcript, timestamps were generated for every word based on force-alignments 3 of audio inputs, and continuous emotional valence ratings were col-lected by annotators 4 .…”

Section: Datasetmentioning

confidence: 99%

Disentangling Latent Emotions of Word Embeddings on Complex Emotional Narratives

Jiang

2019

Natural Language Processing and Chinese Computing

Self Cite

View full text Add to dashboard Cite

Word embedding models such as GloVe are widely used in natural language processing (NLP) research to convert words into vectors. Here, we provide a preliminary guide to probe latent emotions in text through GloVe word vectors. First, we trained a neural network model to predict continuous emotion valence ratings by taking linguistic inputs from Stanford Emotional Narratives Dataset (SEND). After interpreting the weights in the model, we found that only a few dimensions of the word vectors contributed to expressing emotions in text, and words were clustered on the basis of their emotional polarities. Furthermore, we performed a linear transformation that projected high dimensional embedded vectors into an emotion space. Based on NRC Emotion Lexicon (EmoLex), we visualized the entanglement of emotions in the lexicon by using both projected and raw GloVe word vectors. We showed that, in the proposed emotion space, we were able to better disentangle emotions than using raw GloVe vectors alone. In addition, we found that the sum vectors of different pairs of emotion words successfully captured expressed human feelings in the EmoLex. For example, the sum of two embedded word vectors expressing Joy and Trust which express Love shared high similarity (similarity score .62) with the embedded vector expressing Optimism. On the contrary, this sum vector was dissimilar (similarity score -.19) with the the embedded vector expressing Remorse. In this paper, we argue that through the proposed emotion space, arithmetic of emotions is preserved in the word vectors. The affective representation uncovered in emotion vector space could shed some light on how to help machines to disentangle emotion expressed in word embeddings.

show abstract

“…A self-attention layer was used to learn the alignment weights between speech frames and text words from different time-stamps. In addition, Wu et.al., in [158], employed transformer-based self-attention to attend the emotional autobiographical narratives. In their study, attention mechanisms were found to be powerful in a combination of Memory Fusion Network for multimodal fusion of audio, video, and text modalities.…”

Section: Related Work 721 Attention Mechanisms For Multimodal Learningmentioning

confidence: 99%

Bimodal emotion recognition through audio-visual cues

Ghaleb

View full text Add to dashboard Cite

People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.• The final author version and the galley proof are versions of the publication after peer review.• The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User Agreement:

show abstract

Attending to Emotional Narratives

Cited by 14 publications

References 38 publications

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Disentangling Latent Emotions of Word Embeddings on Complex Emotional Narratives

Bimodal emotion recognition through audio-visual cues

Contact Info

Product

Resources

About