Neural sources of prediction errors detect unrealistic VR interactions

Abstract

Objective Neural interfaces hold significant promise to implicitly track user experience. Their application in VR/AR simulations is especially favorable as it allows user assessment without breaking the immersive experience. In VR, designing immersion is one key challenge. Subjective questionnaires are the established metrics to assess the effectiveness of immersive VR simulations. However, administering such questionnaires requires breaking the immersive experience they are supposed to assess. Approach We present a complimentary metric based on a ERPs. For the metric to be robust, the neural signal employed must be reliable. Hence, it is beneficial to target the neural signal's cortical origin directly, efficiently separating signal from noise. To test this new complementary metric, we designed a reach-to-tap paradigm in VR to probe EEG and movement adaptation to visuo-haptic glitches. Our working hypothesis was, that these glitches, or violations of the predicted action outcome, may indicate a disrupted user experience. Main Results Using prediction error negativity features, we classified VR glitches with ~77\% accuracy. We localized the EEG sources driving the classification and found midline cingulate EEG sources and a distributed network of parieto-occipital EEG sources to enable the classification success. Significance Prediction error signatures from these sources reflect violations of user's predictions during interaction with AR/VR, promising a robust and targeted marker for adaptive user interfaces.

Using event-related brain potentials to evaluate motor-auditory latencies in virtual reality

“…The work of colleagues (2019, 2022) has already demonstrated that this is possible for a different type of ERP modulation: They introduced temporal mismatches between sensory stimuli received by the visual and haptic modalities, and showed that these mismatches are accompanied by prediction-errorrelated ERP components (Gehrke et al, 2019). In a follow-up study, they showed that this ERP effect can actually be used as input for a classifier that detects trials with visuo-haptic mismatches solely based on participants' EEG data (Gehrke et al, 2022). Their ERP-based classification reached an accuracy of 77%, whereas classification based on behavioral data barely exceeded chance level (55%).…”

“…Because the assessment rests on a comparison of ERPs elicited by specific self-versus externally generated events precisely locked in time, the method is robust against unspecific variations in user state. Indeed, prediction-related ERPs have already been suggested as a quality metric for VR interactions (Gehrke et al, 2019(Gehrke et al, , 2022. These authors introduced temporal mismatches between different sensory channels -specifically, between visual and haptic stimuli elicited by a user action.…”

“…We examine whether sensory ERPs elicited by self-generated sounds are modulated by latency delay in the described VR context. Finding such modulation would add to the body of evidence regarding latency effects in sensorimotor prediction (Elijah et al, 2016;Oestreich et al, 2016;Pinheiro et al, 2019;Whitford et al, 2011), and would be promising in terms of using this effect for an unobtrusive evaluation of VR quality (akin to Gehrke et al, 2019Gehrke et al, , 2022.…”

The BeMoBIL Pipeline for automated analyses of multimodal mobile brain and body imaging data

“…Motion events can be an integral analysis aspect of MoBI experiments. For example, they can be used to detect reaction time and movement duration (Gehrke et al, 2022; Jungnickel & Gramann, 2016), they can serve as anchors for time-warping in spectral analysis (Gramann et al, 2021), they can help remove oscillatory gait artifacts (Gwin et al, 2010), or they can help to shed light on the neural basis of oscillatory gait generation (Wagner et al, 2016). To enable this functionality, the pipeline contains two functions: A basic movement onset and offset detector that requires only a single tracked element of any kind, and the advanced detection of relevant gait event markers.…”

“…The detector assumes no trend in the data and thus works on data where the endpoint of a movement is the same as the start point in the relevant channels. This can be for example the yaw orientation of the head to detect rotation movements (Gramann et al, 2021), the position of the hand in a reaching task (Gehrke et al, 2022), or the up/down movement of a foot tracker to detect steps (see next section). The detected event markers and used parameters are stored in the data structure so they can be copied between synchronized datasets of different modalities.…”