Detecting Visuo-Haptic Mismatches in Virtual Reality using the Prediction Error Negativity of Event-Related Brain Potentials

Abstract

Designing immersion is the key challenge in virtual reality; this challenge has driven advancements in displays, rendering and recently, haptics. To increase our sense of physical immersion, for instance, vibrotactile gloves render the sense of touching, while electrical muscle stimulation (EMS) renders forces. Unfortunately, the established metric to assess the effectiveness of haptic devices relies on the user's subjective interpretation of unspecific, yet standardized, questions. Here, we explore a new approach to detect a conflict in visuohaptic integration (e.g., inadequate haptic feedback based on poorly configured collision detection) using electroencephalography (EEG). We propose analyzing event-related potentials (ERPs) during interaction with virtual objects. In our study, participants touched virtual objects in three conditions and received either no haptic feedback, vibration, or vibration and EMS feedback. To provoke a brain response in unrealistic VR interaction, we also presented the feedback prematurely in 25% of the trials. We found that the early negativity component of the ERP (so called prediction error) was more pronounced in the mismatch trials, indicating we successfully detected haptic conflicts using our technique. The results can be understood as a first step towards ERPs as a potential metric of haptic immersion in VR. CCS CONCEPTS • Human-centered computing → Haptic devices; Virtual reality;

I feel a moving crowd surrounds me: Exploring tactile feedback during immersive walking in a virtual crowd

“…Therefore, they were evidently not able to distinguish the mismatched stimuli. This contradicts the findings reported in various other studies concerning mismatched stimuli 76,77 and studies indicating that accurate or logical tactile feedback makes the interaction more realistic compared with illogical tactile feedback 7,78 . However, considering that most of the prior studies that we reviewed provided more discrete visual and haptic interaction compared with this study, our results have gone a step further.…”

Grand Field Challenges for Cognitive Neuroergonomics in the Coming Decade

“…Even though improved mobile brain imaging methods allow for investigating naturalistic interactions with technical systems in the workplace and everyday settings (Gehrke et al, 2019;Wascher et al, 2020), these new methods likely come with potential changes in the extracted neural parameters (Gramann et al, 2018). Such differences have to be described and embedded in a systematic fashion to allow for understanding the theoretical and methodological framework and to foster convergence with results from standard laboratory protocols, leading directly to the third grand field challenge.…”

Hypothesis Formalization: Empirical Findings, Software Limitations, and Design Implications

“…When defining proxy variables, researchers (i) used previously validated measures when available for theoretical and methodological soundness, such as the Barcelona Music Reward Questionnaire (BMRQ) to measure music reward (in [36]), or (ii) developed new measures as a research contribution. For example, in [26], the authors develop an EEGbased measure for "immersiveness" in VR they demonstrated to be superior to previous measures that required halting immersive VR experiences to ask users about immersion. Researchers also sometimes justified choosing proxies based on available data.…”