An Investigation of Various Machine and Deep Learning Techniques Applied in Automatic Fear Level Detection and Acrophobia Virtual Therapy

Abstract

In this paper, we investigate various machine learning classifiers used in our Virtual Reality (VR) system for treating acrophobia. The system automatically estimates fear level based on multimodal sensory data and a self-reported emotion assessment. There are two modalities of expressing fear ratings: the 2-choice scale, where 0 represents relaxation and 1 stands for fear; and the 4-choice scale, with the following correspondence: 0—relaxation, 1—low fear, 2—medium fear and 3—high fear. A set of features was extracted from the sensory signals using various metrics that quantify brain (electroencephalogram—EEG) and physiological linear and non-linear dynamics (Heart Rate—HR and Galvanic Skin Response—GSR). The novelty consists in the automatic adaptation of exposure scenario according to the subject’s affective state. We acquired data from acrophobic subjects who had undergone an in vivo pre-therapy exposure session, followed by a Virtual Reality therapy and an in vivo evaluation procedure. Various machine and deep learning classifiers were implemented and tested, with and without feature selection, in both a user-dependent and user-independent fashion. The results showed a very high cross-validation accuracy on the training set and good test accuracies, ranging from 42.5% to 89.5%. The most important features of fear level classification were GSR, HR and the values of the EEG in the beta frequency range. For determining the next exposure scenario, a dominant role was played by the target fear level, a parameter computed by taking into account the patient’s estimated fear level.

Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

“…In recent years, EEG use has increased, with 6 papers being published (14.3%), and the CNS has started to be used, in combination with HMDs, to recognise emotions. The analyses that have been used are ERP [ 138 ], power spectral density [ 140 ] and functional connectivity [ 65 ]. EMG (11.9%) and RSP (9.5) were also used, mostly in combination with HRV.…”

“…The vast majority analysed the implicit responses of the subjects in different emotional states using hypothesis testing (83.33%), correlations (14.29) or linear regression (4.76%). However, in recent years, we have seen the introduction of applied supervised machine-learning algorithms (11.90%), such as SVM [ 105 ], Random Forest [ 139 ] and kNN [ 140 ] to perform automatic emotion recognition models. They have been used in combination with EEG [ 65 ], HRV [ 105 ] and EDA [ 140 ].…”

“…From 2016, the research started to include HRV frequency domain and non-linear domain analyses [ 105 , 119 ], and EDA analyses, such as CDA, dividing the signals into tonic and phasic components [ 64 ]. In terms of the CNS, EEG research has been undertaken since 2016, including ERP [ 138 ], power spectral density [ 140 ] and functional connectivity analysis [ 65 ]. Other non-physiological implicit measures have been used since 2019, such as eye-tracking [ 141 ], gait patterns [ 135 ], navigation [ 133 ] and salivary cortisol responses [ 132 ].…”

“…This procedure collected physiological responses, that is, EDA, HRV, EMG and RSP. Bălan et al [ 140 ] analysed the performance of a set of machine-learning and deep-learning techniques (kNN, SVM, RF, LDA, NN), which adapted their stimuli based on the level of fear recognised, in fear recognition in a 3D acrophobia game. The results showed recognition levels ranging from 42.5% to 89.5%.…”

Attentional threat biases and their role in anxiety: A neurophysiological perspective

“…As a consequence, there is an urgent need to improve diagnostic assessment in a fashion that is (i) oriented towards identifying novel treatment targets while also (ii) providing mechanistic information regarding the specific etiology and symptoms observed in a given patient. Basing diagnostic assessment on biobehavioral markers and crossdiagnostic mechanisms may well open avenues towards these goals, with the long-term aim of personalized psychiatry, in which tailored interventions are offered based on a patient's neurobehavioral profile (Bălan et al, 2020). Neurophysiological markers of attentional bias to threat are promising objects of study in this context.…”