A VR Game-based System for Multimodal Emotion Data Collection

Bassano, Chiara; Ballestin, Giorgio; Ceccaldi, Eleonora; Larradet, Fanny; Mancini, Maurizio; Volta, Erica; Niewiadomski, Radosław

doi:10.1145/3359566.3364695

Cited by 11 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternatively, a study used virtual reality games to assess the CPM across various emotions (Meuleman and Rudrauf, 2018) and found that fear and joy were predicted by appraisal variables better than by other components, whereas these two emotions were generally poorly classified in our study. Other recent studies have also made use of (virtual reality) video games to assess appraisal and other emotion components during brain imaging (Leitão et al, 2020) or physiological (Bassano et al, 2019) measurements.…”

Section: Discussionmentioning

confidence: 99%

Emotion Recognition in a Multi-Componential Framework: The Role of Physiology

Menétrey

Mohammadi

Leitão

et al. 2022

Front. Comput. Sci.

View full text Add to dashboard Cite

The Component Process Model is a well-established framework describing an emotion as a dynamic process with five highly interrelated components: cognitive appraisal, expression, motivation, physiology and feeling. Yet, few empirical studies have systematically investigated discrete emotions through this full multi-componential view. We therefore elicited various emotions during movie watching and measured their manifestations across these components. Our goal was to investigate the relationship between physiological measures and the theoretically defined components, as well as to determine whether discrete emotions could be predicted from the multicomponent response patterns. By deploying a data-driven computational approach based on multivariate pattern classification, our results suggest that physiological features are encoded within each component, supporting the hypothesis of a synchronized recruitment during an emotion episode. Overall, while emotion prediction was higher when classifiers were trained with all five components, a model without physiology features did not significantly reduce the performance. The findings therefore support a description of emotion as a multicomponent process, in which emotion recognition requires the integration of all the components. However, they also indicate that physiology per se is the least significant predictor for emotion classification among these five components.

show abstract

Section: Discussionmentioning

confidence: 99%

Emotion Recognition in a Multi-Componential Framework: The Role of Physiology

Menétrey

Mohammadi

Leitão

et al. 2022

Front. Comput. Sci.

View full text Add to dashboard Cite

show abstract

“…The same appraisal manipulations are addressed by van Reekum et al (2004) in the context of a simple video game to study physiological reactions. In Bassano et al (2019) a Virtual Reality (VR) game and a software platform collecting the player's multimodal data, synchronized with the VR content, are used to build a dataset. The game used was designed according to the emotion elicitation process described by Roseman's appraisal theory.…”

Section: Related Work Applications Of Appraisal Theoriesmentioning

confidence: 99%

“…While our main aim is to introduce a new self-assessment method in the wild, we also believe that we need to perform more controlled experiments. The different methods of self-assessment can be compared when events (stimuli) that potentially may elicit emotional states are experimentally controlled (e.g., in the virtual environment, see a preliminary work by Bassano et al, 2019). This would allow us to compare the reports provided by different participants about the specific events.…”

Section: Figurementioning

confidence: 99%

“…This allows experimenters to control the stimuli and reduce the number of contextual factors that may influence the subjects' reactions. Different types of stimuli are used such as sounds, images and videos (Miranda-Correa et al, 2021) but also more complex techniques including virtual reality experience (Chirico et al, 2018;Marín-Morales et al, 2020;Dozio et al, 2022), playing various types of games (Niewiadomski et al, 2013;Bassano et al, 2019), and interaction with social robots (Redondo et al, 2023). To this date, more rare are studies that have attempted to build real-life (not induced) emotions datasets, i.e., collections of affect-related data, outside of the lab, in reaction to every-day events.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-assessment of affect-related events for physiological data collection in the wild based on appraisal theories

Niewiadomski,

Larradet,

Barresi

et al. 2024

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

This paper addresses the need for collecting and labeling affect-related data in ecological settings. Collecting the annotations in the wild is a very challenging task, which, however, is crucial for the creation of datasets and emotion recognition models. We propose a novel solution to collect and annotate such data: a questionnaire based on the appraisal theory, that is accessible through an open-source mobile application. Our approach exploits a commercially available wearable physiological sensor connected to a smartphone. The app detects potentially relevant events from the physiological data, and prompts the users to report their emotions using a novel questionnaire based on the Ortony, Clore, and Collins (OCC) Model. The questionnaire is designed to gather information about the appraisal process concerning the significant event. The app guides a user through the reporting process by posing a series of questions related to the event. As a result, the annotated data can be used, e.g., to develop emotion recognition models. In the paper, we analyze users' reports. To validate the questionnaire, we asked 22 individuals to use the app and the sensor for a week. The analysis of the collected annotations shed new light on self-assessment in terms of appraisals. We compared a proposed method with two commonly used methods for reporting affect-related events: (1) a two-dimensional model of valence and arousal, and (2) a forced-choice list of 22 labels. According to the results, appraisal-based reports largely corresponded to the self-reported values of arousal and valence, but they differed substantially from the labels provided with a forced-choice list. In the latter case, when using the forced-choice list, individuals primarily selected labels of basic emotions such as anger or joy. However, they reported a greater variety of emotional states when using appraisal theory for self-assessment of the same events. Thus, proposed approach aids participants to focus on potential causes of their states, facilitating more precise reporting. We also found that regardless of the reporting mode (mandatory vs. voluntary reporting), the ratio between positive and negative reports remained stable. The paper concludes with a list of guidelines to consider in future data collections using self-assessment.

show abstract

“…Methods requiring active participation of subjects were also used, e.g., by using video games (Tognetti et al, 2010 ), virtual reality (Ververidis et al, 2008 ) and more recently immersive VR-games (Bassano et al, 2019 ). The latter uses a system composed of a VR-game and software platform to collect the player's physiological data.…”

Section: Common Emotion Elicitation Techniques Used For Data Collementioning

confidence: 99%

Toward Emotion Recognition From Physiological Signals in the Wild: Approaching the Methodological Issues in Real-Life Data Collection

et al. 2020

Self Cite

View full text Add to dashboard Cite

Emotion, mood, and stress recognition (EMSR) has been studied in laboratory settings for decades. In particular, physiological signals are widely used to detect and classify affective states in lab conditions. However, physiological reactions to emotional stimuli have been found to differ in laboratory and natural settings. Thanks to recent technological progress (e.g., in wearables) the creation of EMSR systems for a large number of consumers during their everyday activities is increasingly possible. Therefore, datasets created in the wild are needed to insure the validity and the exploitability of EMSR models for real-life applications. In this paper, we initially present common techniques used in laboratory settings to induce emotions for the purpose of physiological dataset creation. Next, advantages and challenges of data collection in the wild are discussed. To assess the applicability of existing datasets to real-life applications, we propose a set of categories to guide and compare at a glance different methodologies used by researchers to collect such data. For this purpose, we also introduce a visual tool called Graphical Assessment of Real-life Application-Focused Emotional Dataset (GARAFED). In the last part of the paper, we apply the proposed tool to compare existing physiological datasets for EMSR in the wild and to show possible improvements and future directions of research. We wish for this paper and GARAFED to be used as guidelines for researchers and developers who aim at collecting affect-related data for real-life EMSR-based applications.

show abstract

A VR Game-based System for Multimodal Emotion Data Collection

Cited by 11 publications

References 9 publications

Emotion Recognition in a Multi-Componential Framework: The Role of Physiology

Emotion Recognition in a Multi-Componential Framework: The Role of Physiology

Self-assessment of affect-related events for physiological data collection in the wild based on appraisal theories

Toward Emotion Recognition From Physiological Signals in the Wild: Approaching the Methodological Issues in Real-Life Data Collection

Contact Info

Product

Resources

About