Exploiting Electrophysiological Measures of Semantic Processing for Auditory Attention Decoding

Abstract

In Auditory Attention Decoding, a user's electrophysiological brain responses to certain features of speech are modelled and subsequently used to distinguish attended from unattended speech in multi-speaker contexts. Such approaches are frequently based on acoustic features of speech, such as the auditory envelope. A recent paper shows that the brain's response to a semantic description (i.e., semantic dissimilarity) of narrative speech can also be modelled using such an approach. Here we use the (publicly available) data accompanying that study, in order to investigate whether combining this semantic dissimilarity feature with an auditory envelope approach improves decoding performance over using the envelope alone. We analyse data from their 'Cocktail Party' experiment in which 33 subjects attended to one of two simultaneously presented audiobook narrations, for 30 1-minute fragments. We find that the addition of the dissimilarity feature to an envelope-based approach significantly increases accuracy, though the increase is marginal (85.4% to 86.6%). However, we subsequently show that this dissimilarity feature, in which the degree of dissimilarity of the current word with regard to the previous context is tagged to the onsets of each content word, can be replaced with a binary content-word-onset feature, without significantly affecting the results (i.e., modelled responses or accuracy), putting in question the added value of the dissimilarity information for the approach introduced in this recent paper.

Neural markers of speech comprehension: measuring EEG tracking of linguistic speech representations, controlling the speech acoustics

“…This suggests that semantic dissimilarity may explain a response to acoustic properties of the speech rather than a response to language processing. In line with our results, Dijkstra et al (2020) reported that no added value of semantic dissimilarity was seen after controlling for content word onsets.…”

“…To address this discrepancy, a more detailed analysis was performed, which showed that semantic dissimilarity does provide added value over and beyond content words, but fails to do so when also controlling for acoustic and speech segmentation features. Similarly, Dijkstra et al (2020) reported that no added value of semantic dissimilarity was seen after controlling for the acoustic envelope and content word onsets. On the other hand, brain responses seem to be sensitive to semantic dissimilarity under some conditions, as a study of sentence reading found N400-like effects of semantic dissimilarity (Frank and Willems, 2017), and a parallel fMRI investigation, where the participant listened to fragments of audiobooks, localized activity correlated with semantic dissimilarity in non-auditory brain areas (Frank and Willems, 2017).…”

More than Words: Neurophysiological Correlates of Semantic Dissimilarity Depend on Comprehension of the Speech Narrative

“…Together, these findings outline the importance of a coherent context for the comprehension of words and full speech excerpts. Crucially, our semantic dissimilarity measure was sensitive to speech comprehension, giving further support to the conclusion that the neural measure reflects contextual effects on language understanding (see Fig S1 (Broderick et al, 2018)) and refuting the recent suggestions that the semantic dissimilarity TRF may reflect the processing of content words more generally (Dijkstra et al, 2020). In addition, output measures from the semantic dissimilarity TRF were correlated with behavioural scores at the level of individual subjects and individual trials.…”

“…Understanding a word's meaning following the coherent linguistic context in which it appears forms the basis for natural speech comprehension. Recent studies have attempted to obtain electrophysiological indices of this process by modelling how brain responses are affected by a word's meaning relative to its context (Broderick, Anderson, Di Liberto, Crosse, & Lalor, 2018;Dijkstra, Desain, & Farquhar, 2020;Frank & Willems, 2017). One particular approach involved regressing electroencephalographic responses to natural speech against the semantic dissimilarity of words to their preceding context (Broderick et al, 2018).…”

Amplified cortical tracking of word-level features of continuous competing speech in older adults

“…It is, therefore, possible that their estimated TRFs may have captured contributions from other features time-locked to word onsets (e.g., ones related to lexical and syntactic processing). Indeed, in a recent reanalysis of cocktail party data from Broderick et al (2018), Dijkstra et al (2020) showed that replacing dissimilarity values in a regressor with unit-amplitude impulses leads to estimation of essentially identical TRFs to those obtained with the impulses scaled by dissimilarity features. This insensitivity to impulse scaling calls into question the extent to which said TRFs reflect dissimilarity-related processing.…”