Toward FRP-Based Brain-Machine Interfaces—Single-Trial Classification of Fixation-Related Potentials

Abstract

The co-registration of eye tracking and electroencephalography provides a holistic measure of ongoing cognitive processes. Recently, fixation-related potentials have been introduced to quantify the neural activity in such bi-modal recordings. Fixation-related potentials are time-locked to fixation onsets, just like event-related potentials are locked to stimulus onsets. Compared to existing electroencephalography-based brain-machine interfaces that depend on visual stimuli, fixation-related potentials have the advantages that they can be used in free, unconstrained viewing conditions and can also be classified on a single-trial level. Thus, fixation-related potentials have the potential to allow for conceptually different brain-machine interfaces that directly interpret cortical activity related to the visual processing of specific objects. However, existing research has investigated fixation-related potentials only with very restricted and highly unnatural stimuli in simple search tasks while participant’s body movements were restricted. We present a study where we relieved many of these restrictions while retaining some control by using a gaze-contingent visual search task. In our study, participants had to find a target object out of 12 complex and everyday objects presented on a screen while the electrical activity of the brain and eye movements were recorded simultaneously. Our results show that our proposed method for the classification of fixation-related potentials can clearly discriminate between fixations on relevant, non-relevant and background areas. Furthermore, we show that our classification approach generalizes not only to different test sets from the same participant, but also across participants. These results promise to open novel avenues for exploiting fixation-related potentials in electroencephalography-based brain-machine interfaces and thus providing a novel means for intuitive human-machine interaction.

EEG Negativity in Fixations Used for Gaze-Based Control: Toward Converting Intentions into Actions with an Eye-Brain-Computer Interface

“…Such contamination cause problems either when the EOG artifacts appear in the time intervals that are the main focus of analysis or when they are present during intervals used for baseline correction (Finke et al, 2016; Nikolaev et al, 2016). In addition, EEG components within the first 200 ms after fixation onset depend on the size of saccade preceding the fixation (Nikolaev et al, 2016).…”

“…The gaze fixation and FRP data complement each other when used as an input to a hybrid interface: the fixation onset provides the temporal marker required for EEG segmentation and FRP extraction, the FRP provides features for a passive BCI classifier that automatically recognizes target finding (in the case of visual search) or issuing a command (in intentional dwell-based control), and the gaze position indicates the target (Protzak et al, 2013; Finke et al, 2016). This combination may enable fluent interaction both in visual search (Finke et al, 2016; see also Introduction) and in intentional dwell-based control (Protzak et al, 2013, and the current study).…”

“…Recent studies of the FRPs in the visual search tasks demonstrated that finding an object, area or position of interest can be detected using a combination of eye tracking and EEG (Healy and Smeaton, 2011; Kamienkowski et al, 2012; Brouwer et al, 2013; Kaunitz et al, 2014; Devillez et al, 2015; Finke et al, 2016; Ušćumlić and Blankertz, 2016; Wenzel et al, 2016). In particular, Finke et al (2016) reported high performance, with area under ROC curve (AUC) of about 0.9, for classification of short target vs. non-target fixations. Similar phenomena could be a basis of classification in studies by Ihme and Zander (2011) and Protzak et al (2013).…”

Is Neural Activity Detected by ERP-Based Brain-Computer Interfaces Task Specific?

“…However, it can be expected that several words or pictograms are shown in parallel in a more realistic setting. The combination of EEG with an eye tracker would allow to relate the neural activity to each pictogram or word [15–20]. Eye movements towards the items could be used as time points of reference for the EEG segmentation in epochs, instead of the onset of stimuli popping up on the screen.…”

“…Recently, it was suggested that BCI technology could be transferred to relevance detection in human-computer interaction (HCI), because EEG combined with an eye tracker can be used to predict which of several items displayed at the same time on the screen are task-relevant for the user [15–20]. The application of BCI technology to HCI is presumably most useful and convenient for the user if the information (e.g.…”