Speech-based Continuous Emotion Prediction by Learning Perception Responses related to Salient Events

Gamage, Kalani Wataraka; Dang, Ting; Sethu, Vidhyasaharan; Epps, Julien; Ambikairajah, Eliathamby

doi:10.1145/3266302.3266314

Cited by 8 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The presented results demonstrate that the proposed CNN architecture outperforms the LSTM architecture in all tasks, except for arousal on the Development and the Hungarian Test set, where matching results are achieved. While it could be argued that the hyperparameters of the LSTM model allow for more tuning, also the multi-modal baselines of AVEC and all tuned systems based on only the audio modality are outperformed on the German Test set for arousal [22,25,27]. For valence, our system is superior compared to three other models, considering the intra-cultural evaluation.…”

Section: Discussionmentioning

confidence: 86%

“…Time-dependent modelling was widely proposed in the contributions to AVEC. In the approach by Wataraka Gamage et al, emotional dimensions are modelled as the outputs of time-invariant filter arrays, each filter representing a 'salient event' [22]. Huang et al employ a fusion of different (handcrafted and deep) feature sets and an LSTM-RNN [14] and investigate on data augmentation by cutting and overlapping the long sequences.…”

Section: Corpusmentioning

confidence: 99%

See 1 more Smart Citation

Continuous Emotion Recognition in Speech — Do We Need Recurrence?

Schmitt¹,

Cummins²,

Schuller³

2019

Interspeech 2019

View full text Add to dashboard Cite

Emotion recognition in speech is a meaningful task in affective computing and human-computer interaction. As human emotion is a frequently changing state, it is usually represented as a densely sampled time series of emotional dimensions, typically arousal and valence. For this, recurrent neural network (RNN) architectures are employed by default when it comes to modelling the contours with deep learning approaches. However, the amount of temporal context required is questionable, and it has not yet been clarified whether the consideration of long-term dependencies is actually beneficial. In this contribution, we demonstrate that RNNs are not necessary to accomplish the task of time-continuous emotion recognition. Indeed, results gained indicate that deep neural networks incorporating less complex convolutional layers can provide more accurate models. We highlight the pros and cons of recurrent and nonrecurrent approaches and evaluate our methods on the public SEWA database, which was used as a benchmark in the 2017 and 2018 editions of the AudioVisual Emotion Challenge.

show abstract

Section: Discussionmentioning

confidence: 86%

Section: Corpusmentioning

confidence: 99%

Continuous Emotion Recognition in Speech — Do We Need Recurrence?

Schmitt¹,

Cummins²,

Schuller³

2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…Interestingly, approaches in the AVEC 2018 CES did not employ approaches such as transfer learning [80,81] or domain adaptation techniques [29,54] typically seen in cross-cultural testing. In [76], the authors proposed a model based on emotional salient detection to identify emotion markers invariant to sociocultural context. The other two entrants employed data driven approaches based on long short-term memory recurrent neural networks (LSTM-RNN) [27,82].…”

Section: Cross-cultural Emotion Recognitionmentioning

confidence: 99%

AVEC 2018 Workshop and Challenge

Ringeval

Schuller

Valstar

et al. 2018

Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop

127

View full text Add to dashboard Cite

The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) "State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition" is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly under the same conditions. The goal of the Challenge is to provide a common benchmark test set for multimodal information processing and to bring together the health and emotion recognition communities, as well as the audiovisual processing communities, to compare the relative merits of various approaches to health and emotion recognition from reallife data. This paper presents the major novelties introduced this year, the challenge guidelines, the data used, and the performance of the baseline systems on the three proposed tasks: state-of-mind *

show abstract

“…Finally, we end our review by mentioning one more alternative class of models: event-based models such as point-process models [76], [77], [78], [79], which aim to model the time and intensity of events and their impact on a dependent variable. Although event-based approaches are not common within affective computing, one notable and recent example is [80], who proposed an event-filter model to predict valence and arousal over time from speech events. In their model, a vocal event j, occuring at times predicted by ϕ j (t), produces an emotional "response" h j (t).…”

Section: Integrating Discriminative and Generative Approachesmentioning

confidence: 99%

“…Then, the emotional signal Y (t) is then proportional to the sum of the convolution h j (t) ϕ j (t) across all events j. They tested their eventfilter model on the AVEC 2018 dataset, and it performed better than the audio-channel-only baselines for the AVEC 2017 and 2018 challenges [80]. While we do not go further into event-based models in this paper, we do think it offers an alternative approach to modeling time-series emotions, which should be explored more in future research.…”

Section: Integrating Discriminative and Generative Approachesmentioning

confidence: 99%

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Ong

Zhi-Xuan

et al. 2021

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. We demonstrate several baseline and state-of-the-art modeling approaches on the SEND, including a Long Short-Term Memory model and a multimodal Variational Recurrent Neural Network, which perform comparably to the human-benchmark. We end by discussing the implications for future research in time-series affective computing.recognition. Specifically, we define time-series modeling as taking in temporally continuous input data and producing temporally continuous output, with an explicit consideration of how information is propagated over time. For instance, in order to engage in such inference, a social robot in conversation with its user would have to take in a continuous stream of sensor data, process them, and reason about their user's emotions over time, perhaps after every second or after every sentence, as well as across many sentences in the conversation and across multiple conversations [11].Despite the progress that has been made in time-series emotion recognition in the past decade, the field is still far from affective robots that can understand human emotions in daily life. What is needed to achieve this ambitious goal? We suggest that the biggest barriers to overcome are due to (1) the inherent difficulty of building computational time-series models, and (2) the difficulty of collecting highquality datasets. To address this first gap, we conduct a review covering different machine-learning-based approaches to time-series modeling (Section 2). We begin by discussing the most common time-series techniques in affective computing: deep neural network models, part of a broader class of discriminative models. We also cover generative time-series approaches, which are comparatively less popular within affective computing, but offer interesting modeling capabilities and hold exciting potential for emotion understanding.We turn next to discuss the second gap: Researchers need high-quality time-series datasets on which to train models. These are expensive to construct, in terms of both the production of stimuli and the collection of timeseries annotations of emotion and affective labeling [12]. There are several existing time-series datasets that have been used by the ...

show abstract

Speech-based Continuous Emotion Prediction by Learning Perception Responses related to Salient Events

Cited by 8 publications

References 14 publications

Continuous Emotion Recognition in Speech — Do We Need Recurrence?

Continuous Emotion Recognition in Speech — Do We Need Recurrence?

AVEC 2018 Workshop and Challenge

Modeling Emotion in Complex Stories: The Stanford Emotional Narratives Dataset

Contact Info

Product

Resources

About