Towards a Better Gold Standard

Abstract

Emotions are often perceived by humans through a series of multimodal cues, such as verbal expressions, facial expressions and gestures. In order to recognise emotions automatically, reliable emotional labels are required to learn a mapping from human expressions to corresponding emotions. Dimensional emotion models have become popular and have been widely applied for annotating emotions continuously in the time domain. However, the statistical relationship between emotional dimensions is rarely studied. This paper provides a solution to automatic emotion recognition for the Audio/Visual Emotion Challenge (AVEC) 2018. The objective is to find a robust way to detect emotions using more reliable emotion annotations in the valence and arousal dimensions. The two main contributions of this paper are: 1) the proposal of a new approach capable of generating more dependable emotional ratings for both arousal and valence from multiple annotators by extracting consistent annotation features; 2) the exploration of the valence and arousal distribution using outlier detection methods, which shows a specific oblique elliptic shape. With the learned distribution, we are able to detect the prediction outliers based on their local density deviations and correct them towards the learned distribution. The proposed method performance is evaluated on the RECOLA database containing audio, video and physiological recordings. Our results show that a moving average filter is sufficient to remove the incidental errors in annotations. The unsupervised dimensionality reduction approaches could be used to determine a gold standard annotations from multiple annotations. Compared with the baseline model of AVEC 2018, our approach improved the arousal and valence prediction of concordance correlation coefficient significantly to respectively 0.821 and 0.589.

Toward Explainable Affective Computing: A Review

“…Understanding the target distribution can also help improve the reliability of annotations. Wang et al [58] explore the distribution of emotion annotations using outlier detection methods and use these insights to correct outliers toward the learned distribution, reducing labeling noise and outperforming previous SOTA results. Escalante et al.…”

Embracing and Exploiting Annotator Emotional Subjectivity: An Affective Rater Ensemble Model

“…Ringeval et al used median filtering with a window width of three samples before creating a single gold standard using EWE [21]. In a 2018 emotion challenge aiming at improving the gold-standard [23], Wang et al argued that secondary, slight errors in annotations can be removed by a moving average filter [24]. While testing three different filtering techniques to smooth annotation data (Savitzky-Golay filter, moving average filter, and median filter) Thammasan et al found that the moving average filter is more practical to enhance emotion recognition performance [25].…”