An LMMSE-based Estimation of Temporal Response Function in Auditory Attention Decoding

Kuruvila, Ivine; Fischer, Eghart; Hoppe, Ulrich

doi:10.1109/embc44109.2020.9175866

Cited by 4 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…State-of-the-art AAD algorithms are based on linear systems theory where acoustic features are linearly mapped on to the EEG signals. This mapping can be either in the forward direction (Lalor and Foxe, 2010 ; Fiedler et al, 2017 ; Kuruvila et al, 2020 ) or in the backward direction (O'Sullivan et al, 2014 ; Mirkovic et al, 2015 ; Biesmans et al, 2017 ). These algorithms have been successful in providing insights into the underlying neuroscientific processes through which brain suppresses the ignored speaker in a dual-speaker scenario.…”

Section: Introductionmentioning

confidence: 99%

Extracting the Auditory Attention in a Dual-Speaker Scenario From EEG Using a Joint CNN-LSTM Model

et al. 2021

Self Cite

View full text Add to dashboard Cite

Human brain performs remarkably well in segregating a particular speaker from interfering ones in a multispeaker scenario. We can quantitatively evaluate the segregation capability by modeling a relationship between the speech signals present in an auditory scene, and the listener's cortical signals measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids where the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are based on linear systems theory where cues such as speech envelopes are mapped on to the EEG signals. Here, we present a joint convolutional neural network (CNN)—long short-term memory (LSTM) model to infer the auditory attention. Our joint CNN-LSTM model takes the EEG signals and the spectrogram of the multiple speakers as inputs and classifies the attention to one of the speakers. We evaluated the reliability of our network using three different datasets comprising of 61 subjects, where each subject undertook a dual-speaker experiment. The three datasets analyzed corresponded to speech stimuli presented in three different languages namely German, Danish, and Dutch. Using the proposed joint CNN-LSTM model, we obtained a median decoding accuracy of 77.2% at a trial duration of 3 s. Furthermore, we evaluated the amount of sparsity that the model can tolerate by means of magnitude pruning and found a tolerance of up to 50% sparsity without substantial loss of decoding accuracy.

show abstract

Section: Introductionmentioning

confidence: 99%

Extracting the Auditory Attention in a Dual-Speaker Scenario From EEG Using a Joint CNN-LSTM Model

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…where E(•) corresponds the sample mean, C rr corresponds to the autocovariance of the observation signal and C θ r corresponds to the cross-covariance between the observation and the system response. Equation ( 3) can be further expanded as [18] θ…”

Section: Attention Decoding Frameworkmentioning

confidence: 99%

“…1 depicts the SNR distribution of the AEPs obtained at different scalp locations. SNRs are usually between -9 dB to -17 dB and electrodes closer to the reference electrode have low SNR compared to the vertex electrodes [18] [30]. Consequently, we can use signals at electrodes closest to the reference electrode to calculate the covariance matrix of noise that is required to solve (4) or (6).…”

Section: Attention Decoding Frameworkmentioning

confidence: 99%

“…In [18], initialization parameters θ [−1] and M[−1] were chosen as zero mean and unit variance respectively. However, we decided to use the first block to calculate θ [−1] and M[−1].…”

Section: Estimating Model Parametersmentioning

confidence: 99%

“…The first module relates to the dynamic estimation of the TRF corresponding to the attended speaker (attended TRF) and the unattended speaker (unattended TRF). It is based on sequential linear minimum mean squared error (LMMSE) estimator which is an improvement of the algorithm proposed in [18]. LMMSE algorithms are based on explicitly calculating the covariance of the signal component and subsequently applying Bayesian estimation theory.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Inference of the Selective Auditory Attention using Sequential LMMSE Estimation

Kuruvila¹,

Demir²,

Fischer³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Attentive listening in a multispeaker environment such as a cocktail party requires suppression of the interfering speakers and the noise around. People with normal hearing perform remarkably well in such situations. Analysis of the cortical signals using electroencephalography (EEG) has revealed that the EEG signals track the envelope of the attended speech stronger than that of the interfering speech. This has enabled the development of algorithms that can decode the selective attention of a listener in controlled experimental settings. However, often these algorithms require longer trial duration and computationally expensive calibration to obtain a reliable inference of attention. In this paper, we present a novel framework to decode the attention of a listener within trial durations of the order of two seconds. It comprises of three modules: 1) Dynamic estimation of the temporal response functions (TRF) in every trial using a sequential linear minimum mean squared error (LMMSE) estimator, 2) Extract the N1−P2 peak of the estimated TRF that serves as a marker related to the attentional state and 3) Obtain a probabilistic measure of the attentional state using a support vector machine followed by a logistic regression. The efficacy of the proposed decoding framework was evaluated using EEG data collected from 27 subjects. The total number of electrodes required to infer the attention was four: One for the signal estimation, one for the noise estimation and the other two being the reference and the ground electrodes. Our results make further progress towards the realization of neuro-steered hearing aids.

show abstract