A functional data analysis approach for continuous 2-D emotion annotations

Abstract

The standard paradigm in Affective Computing involves acquiring one/several markers (e.g., physiological signals) of emotions and training models on these to predict emotions. However, due to the internal nature of emotions, labelling/annotation of emotional experience is done manually by humans using specially developed annotation tools. To effectively exploit the resulting subjective annotations for developing affective systems, their quality needs to be assessed. This entails, (i) evaluating the variations in annotations, across different subjects and emotional stimuli, to detect spurious/unexpected patterns; and (ii) developing strategies to effectively combine these subjective annotations into a ground truth annotation. This article builds on our previous work by presenting a novel Functional Data Analysis based approach to assess the quality of annotations. Specifically, the bivariate annotation time-series are transformed into functions, such that each resulting functional annotation then becomes a sample element for analysis like Multivariate Functional Principal Component Analysis (MFPCA) that evaluate variation across all annotations. The resulting scores from MFPCA provide interesting insights into annotation patterns and facilitate the use of multivariate statistical techniques to address both (i) and (ii). Given the presented efficacy of these methods, we believe they offer an exciting new approach to assessing the quality of annotations.

A dataset of continuous affect annotations and physiological signals for emotion analysis

“…The quality and the reliability of the annotation data has been thoroughly addressed in our previous works [27,28,16,29]. A summary of the relevant highlights from these works is presented below.…”

“…According to the ratings received on the same, the annotation setup had 'excellent' usability as the participants found it to be consistent, intuitive and simple to use [16]. In [16,29], several different methods for analysing the annotation patterns in continuous 2D annotations were presented. The results of these continuous methods were concurrent to the results of the MANOVA.…”

“…These being, that separate annotation of valence and arousal doesn't account for the inherent relationship between these dimensions [23,25], and that mouse-based annotation tools are generally less ergonomic than joysticks [23,24,26,22]. To address these drawbacks, we developed a novel Joystick-based Emotion Reporting Interface (JERI) that facilitates simultaneous annotation of valence and arousal [27,28,16,29]. A testament to the efficacy of JERI is that in recent years, several similar annotation setups have been presented [25,30].…”

“…The physiological measures included in the dataset are from Electrocardiograph (ECG), Blood Volume Pulse (BVP), Galvanic Skin Response (GSR), Respiration (RSP), Skin Temperature (SKT) and Electromyography (EMG) sensors. The annotation data has been previously used for several publications aimed at introducing, analysing and validating this approach to emotion annotation [28,16,29]. However, it has not been previously released.…”

Estimation of Organizational Competitiveness by a Hybrid of One-Dimensional Convolutional Neural Networks and Self-Organizing Maps Using Physiological Signals for Emotional Analysis of Employees

“…While all these approaches have their specific advantages and disadvantages, as Kim and Fesenmaier suggested, physiological measurement is particularly objective [ 37 ]. In this manuscript, different repositories [ 38 , 39 , 40 , 41 ] are used for the input data acquisition. These datasets facilitate generating data for this research.…”