Fast mental states decoding in mixed reality

Abstract

The combination of Brain-Computer Interface (BCI) technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality (MR) systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. Brain states discrimination during mixed reality experience is thus critical for adapting specific data features to contingent brain activity. In this study we recorded electroencephalographic (EEG) data while participants experienced MR scenarios implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the user's actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in MR, using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5 s time-window shifting every 200 ms. Our results indicate overall better performance of LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled MR scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in MR.

Real-time prediction of short-timescale fluctuations in cognitive workload

“…These results were sufficiently encouraging to inspire a great deal of research along similar lines with EEG-based measures, including ERPs based on events associated with secondary tasks such as tone counting and oddball detection (e.g., Sirevaag et al., 1993 ; Kramer et al., 1995 ; Allison and Polich 2008 ). Frequency-based EEG measures, which are attractive because they do not require a link to primary- or secondary-task events, used increasingly sophisticated analysis approaches, such as individual participant multivariate analyses (e.g., Gevins et al., 1998 ) and analyses addressing problems with non-stationarity (e.g., Murata 2005 ), and were applied to a broad array of tasks and settings taking advantage of the possibility of more mobile recording (e.g., De Massari et al., 2014 ; Smith et al., 2001 ; Mijović et al., 2017 ). EEG has been used as a basis for adaptive automation in a closed-loop system (Pope et al., 1995 ; Prinzel et al., 2000 ) with a more recent implementation switching to ERPs because of their greater specificity to cognitive workload (Prinzel et al., 2003 ).…”

On the Use of Machine Learning for EEG-Based Workload Assessment: Algorithms Comparison in a Realistic Task

“…Therefore, the application of Machine Learning has been considered the solution for classifying the workload and overcoming these issues typical of real applications. The preliminary analysis of the works carried out so far in this context has shown that it is possible to discriminate with acceptable accuracy only two levels of workload [6], [18], [22], [24], [25], [27]- [29], [33], [36]- [39], [42], [45], [46], [58], even though, above all in view of a practical application of the workload measurement, it is necessary to establish at least the value of two thresholds to define the underload and the overload state. The most frequently employed features are the spectral ones, because they can be calculated with a high temporal resolution (up to one second) and allow to monitor brain activity in a quantitative manner without temporal triggers.…”

Classification of Affective States via EEG and Deep Learning

“…Massari et al [25] and Kovacevic et al [26], respectively used mixed reality stimuli to conduct brain state recognition based solely on EEG signals as input to the classification system. Massari et al utilized their proprietary eXperience Induction Machine (XIM) as the mixed reality stimuli system to classify different brain states for spatial navigation, reading and calculation, achieving the best results of 86% using linear discriminant analysis [25].…”

“…Massari et al [25] and Kovacevic et al [26], respectively used mixed reality stimuli to conduct brain state recognition based solely on EEG signals as input to the classification system. Massari et al utilized their proprietary eXperience Induction Machine (XIM) as the mixed reality stimuli system to classify different brain states for spatial navigation, reading and calculation, achieving the best results of 86% using linear discriminant analysis [25]. Kovacevic et al implemented an EEG-based mental state recognition system as part of an immersive and interactive multi-media science-art installation using the recognition of relaxation and concentration mental states of its participants to determine the audio-visual output of a dome-based artistic installation comprising video animations that were projected on to the 360° surface of the semitransparent dome as well as the generation of soundscapes based on pre-recorded sound libraries and live improvisations [26].…”