Real-time inference of word relevance from electroencephalogram and eye gaze

Abstract

It was demonstrated that the interest of a reader can be inferred online from EEG and eye tracking signals, which can potentially be used in novel types of adaptive software, which enrich the interaction by adding implicit information about the interest of the user to the explicit interaction. The study is characterised by the following novelties. Interpretation with respect to the word meaning was necessary in contrast to the usual practice in brain-computer interfacing where stimulus recognition is sufficient. The typical counting task was avoided because it would not be sensible for implicit relevance detection. Several words were displayed at the same time, in contrast to the typical sequences of single stimuli. Neural activity was related with eye tracking to the words, which were scanned without restrictions on the eye movements.

The N400 for brain computer interfacing: complexities and opportunities

“…This application of inferring information about the active mental context of the user was the third application we discussed. On one hand there is evidence that this approach can be used successfully to determine which of a small number of categories a subject aims to select ('has on mind') [16]. By using fixation-related potentials, stimuli can be presented to a user at a fast speed, as the inter-stimulus interval is determined by the subject, through their gaze.…”

“…), but a more straightforward application may be to determine appropriate reading levels in children. A study by [94] investigated the N400 in ages between 5 and 26 and found N400 amplitudes to be larger for younger subjects, with amplitudes stabilising around age [15][16]. A comprehension detection application may thus prove especially sensitive when aimed at children.…”

“…This may reflect a disinterest in the kind of BCI paradigms that could be designed around a N400 effect, or, a (perceived) difficulty to make such paradigms work. To obtain more insight, we searched for the keywords "Brain Computer Interface" and "N400" and identified three research lines that do make use of an N400 effect: (1) the use of the N400 in response to familiar face stimuli for boosting performance of the matrix speller [10][11][12], (2) to detect the absence or presence of language processing in patients with impaired levels of consciousness (Disorders of Consciousness) [13][14][15], and (3) to probe for information about the active mental state of the user, e.g., a category or word on the user's mind, by presenting stimuli that elicit responses of relatedness with regard to this mental context [16][17][18].…”

Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural Networks

“…These features are then used for statistical inferences, classification, and prediction. For instance, some variants of aggregated fixation-count and fixation-duration were used in studies reported in [14,15,19,21,39,41,59,63]. Eye-dwell time and/or visit time was used by Fahey et al [14].…”

“…They reported an accuracy of 70%, but did not mention the specific eye-tracking features used. Wenzel et al [59] inferred the relevance of individual words from fixation duration and EEG features. They reported that only using fixation-duration gave an AUC of 0.51 (marginally better than chance), whereas combining fixation-durations with EEG features improved the AUC to 0.63.…”