A Tutorial on Auditory Attention Identification Methods

Abstract

Auditory attention identification methods attempt to identify the sound source of a listener's interest by analyzing measurements of electrophysiological data. We present a tutorial on the numerous techniques that have been developed in recent decades, and we present an overview of current trends in multivariate correlation-based and model-based learning frameworks. The focus is on the use of linear relations between electrophysiological and audio data. The way in which these relations are computed differs. For example, canonical correlation analysis (CCA) finds a linear subset of electrophysiological data that best correlates to audio data and a similar subset of audio data that best correlates to electrophysiological data. Model-based (encoding and decoding) approaches focus on either of these two sets. We investigate the similarities and differences between these linear model philosophies. We focus on (1) correlation-based approaches (CCA), (2) encoding/decoding models based on dense estimation, and (3) (adaptive) encoding/decoding models based on sparse estimation. The specific focus is on sparsity-driven adaptive encoding models and comparing the methodology in state-of-the-art models found in the auditory literature. Furthermore, we outline the main signal processing pipeline for how to identify the attended sound source in a cocktail party environment from the raw electrophysiological data with all the necessary steps, complemented with the necessary MATLAB code and the relevant references for each step. Our main aim is to compare the methodology of the available methods, and provide numerical illustrations to some of them to get a feeling for their potential. A thorough performance comparison is outside the scope of this tutorial.

Decoding the Attended Speaker From EEG Using Adaptive Evaluation Intervals Captures Fluctuations in Attentional Listening

“…Identifying the degree and direction of attention in near realtime requires that this information can be extracted from short time intervals. Several studies have shown that attention can be reliably decoded from single-trial EEG data in the two competing speaker paradigm (Horton et al, 2014;Mirkovic et al, 2015Mirkovic et al, , 2016O'Sullivan et al, 2015;Biesmans et al, 2017;Fiedler et al, 2017;Fuglsang et al, 2017Fuglsang et al, , 2020Haghighi et al, 2017) using various auditory attention decoding (AAD) methods (for a review see: Alickovic et al, 2019). In these studies, AAD procedures demonstrated above chance-level accuracy for evaluation periods of time ranging from 2 to 60 s. In a neurofeedback application, features should be obtained as quickly as possible.…”

Robust EEG-Based Decoding of Auditory Attention With High-RMS-Level Speech Segments in Noisy Conditions

“…Several recent magnetoencephalographic and electroencephalographic (EEG) studies have shown that neural responses during auditory selectivity tasks correlate more strongly with attended than with ignored speech (e.g., Ding and Simon, 2012 ; Horton et al, 2013 ; O’Sullivan et al, 2015 ). Auditory attention decoding models have been established to describe the relationship between continuous speech and ongoing cortical recordings ( Alickovic et al, 2019 ). The linear temporal response function (TRF) model ( Crosse et al, 2016 ) has been used widely to predict EEG responses to speech (i.e., the encoding model; e.g., Di Liberto et al, 2015 ) and to reconstruct speech from associated EEG signals (i.e., the decoding model; e.g., Ding and Simon, 2012 ; Mirkovic et al, 2015 ; Teoh and Lalor, 2019 ) using off-line regression techniques.…”

“…As the relative intensity of background interference may affect the quality of neural tracking of attended speech ( Alickovic et al, 2019 ), the impacts of various SNR conditions on auditory attention decoding in realistic scenarios should also be considered. Generally, low SNRs interfere with attended speech segregation, and the quality of neural tracking of attended speech declines with increasing noise intensity ( Kong et al, 2014 ; Das et al, 2018 ).…”

“…Then, the TRF was used to describe the relationships between EEG signals and the features of attended and ignored speech (i.e., the temporal envelopes of intact speech and high-RMS-level segments alone) under different SNR conditions ( Crosse et al, 2016 ). Auditory attention decoding performance was assessed by examining correlations between reconstructed and actual speech signals ( Alickovic et al, 2019 ). We hypothesized that: (1) the neural-tracking activities reflected by TRF responses with intact temporal envelopes would be stronger than that with high-RMS-level segments, as the high-RMS-level segments only contain a part of acoustic information; (2) the attention decoding accuracy with high-RMS-level speech segments would not be inferior to that with intact speech in the presence of background interference, as the high-RMS-level segments would carry sufficient information for the segregation and perception of attended speech; and (3) the top-down modulation of auditory attention would facilitate the classification between attended and ignored speech based on the cortical tracking ability, and be insensitive to the change of noise level.…”

An Interpretable Performance Metric for Auditory Attention Decoding Algorithms in a Context of Neuro-Steered Gain Control

“…To illustrate why and how the MESD is useful in the evaluation of AAD algorithms, we apply it to an illustrative example in which we compare two variants of the MMSE decoder for AAD as proposed in [6] and [10], respectively. 1) Description of the two variants: Given a training set of M data windows, in the first variant of [6] (also adopted in, e.g., [8]), per-window (corresponding to decision window length τ ) decoders are computed, after which the M decoders are averaged to obtain one final decoder. The second variant of [10] (also adopted in, e.g., [12], [17], [18]), first averages the M per-window autocorrelation matrices (or equivalently: the windows are all concatenated) to train a single decoder across all training windows simultaneously.…”