Comparison of Two-Talker Attention Decoding from EEG with Nonlinear Neural Networks and Linear Methods

Ciccarelli, Gregory; Nolan, Michael A.; Perricone, Joseph; Calamia, Paul; Haro, Stephanie; O’Sullivan, James; Mesgarani, Nima; Quatieri, Thomas F.; Smalt, Christopher J.

doi:10.1101/504522

Cited by 3 publications

(3 citation statements)

References 34 publications

(65 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future work needs to establish the minimum accuracy that would be required for a listener to receive a perceptual benefit from such a device. The use of deep learning for improving attention decoding accuracies has a good track record in scalp-EEG based studies of auditory attention, and the benefits seen in those studies would presumably translate across to wearable EEG studies [41]- [43]. The null distributions of the attention markers are also shown (grey).…”

Section: Discussionmentioning

confidence: 99%

Robust decoding of the speech envelope from EEG recordings through deep neural networks

2022

View full text Add to dashboard Cite

Objective. Smart hearing aids which can decode the focus of a user's attention could considerably improve comprehension levels in noisy environments. Methods for decoding auditory attention from electroencephalography (EEG) have attracted considerable interest for this reason. Recent studies suggest that the integration of deep neural networks (DNNs) into existing auditory attention decoding algorithms is highly beneficial, although it remains unclear whether these enhanced algorithms can perform robustly in different real-world scenarios. To this end, we sought to characterise the performance of DNNs at reconstructing the envelope of an attended speech stream from EEG recordings in different listening conditions. In addition, given the relatively sparse availability of EEG data, we investigate possibility of applying subject-independent algorithms to EEG recorded from unseen individuals. Approach. Both linear models and nonlinear DNNs were employed to decode the envelope of clean speech from EEG recordings, with and without subject-specific information. The mean behaviour, as well as the variability of the reconstruction, was characterised for each model. We then trained subject-specific linear models and DNNs to reconstruct the envelope of speech in clean and noisy conditions, and investigated how well they performed in different listening scenarios. We also established that these models can be used to decode auditory attention in competing-speaker scenarios. Main results. The DNNs offered a considerable advantage over their linear counterpart at reconstructing the envelope of clean speech. This advantage persisted even when subject-specific information was unavailable at the time of training. The same DNN architectures generalised to a distinct dataset, which contained EEG recorded under a variety of listening conditions. In competing-speakers and speech-in-noise conditions, the DNNs significantly outperformed the linear models. Finally, the DNNs offered a considerable improvement over the linear approach at decoding auditory attention in competing-speakers scenarios. Significance. We present the first detailed study into the extent to which DNNs can be employed for reconstructing the envelope of an attended speech stream. We conclusively demonstrate that DNNs have the ability to improve the reconstruction of the attended speech envelope. The variance of the reconstruction error is shown to be similar for both DNNs and the linear model. Overall, DNNs are demonstrated to show promise for real-world auditory attention decoding, since they perform well in multiple listening conditions and generalise to data recorded from unseen participants.

show abstract

Section: Discussionmentioning

confidence: 99%

Robust decoding of the speech envelope from EEG recordings through deep neural networks

2022

View full text Add to dashboard Cite

show abstract

“…Leastsquares Ding and Simon, 2012a;Di Liberto et al, 2015;Alickovic et al, 2016, in rewiev;Fiedler et al, 2017Fiedler et al, , 2019Hjortkjaer et al, 2018;Kalashnikova et al, 2018;Lesenfants et al, 2018;Lunner et al, 2018;Verschueren et al, 2018;Wong et al, 2018 Inverse/backward modeling Supervised case: O'Sullivan et al, 2015O'Sullivan et al, , 2017Aroudi et al, 2016;Das et al, 2016Das et al, , 2018Presacco et al, 2016;Biesmans et al, 2017;Fuglsang et al, 2017;Van Eyndhoven et al, 2017;Zink et al, 2017;Bednar and Lalor, 2018;Ciccarelli et al, 2018;Etard et al, 2018;Hausfeld et al, 2018;Narayanan and Bertrand, 2018;Schäfer et al, 2018;Vanthornhout et al, 2018;Verschueren et al, 2018;Wong et al, 2018;Akbari et al, 2019;Somers et al, 2019 trial separately, and averaging over all training de/en-coders (Crosse et al, 2016).…”

Section: Computational Models In Practicementioning

confidence: 99%

A Tutorial on Auditory Attention Identification Methods

et al. 2019

View full text Add to dashboard Cite

Auditory attention identification methods attempt to identify the sound source of a listener's interest by analyzing measurements of electrophysiological data. We present a tutorial on the numerous techniques that have been developed in recent decades, and we present an overview of current trends in multivariate correlation-based and model-based learning frameworks. The focus is on the use of linear relations between electrophysiological and audio data. The way in which these relations are computed differs. For example, canonical correlation analysis (CCA) finds a linear subset of electrophysiological data that best correlates to audio data and a similar subset of audio data that best correlates to electrophysiological data. Model-based (encoding and decoding) approaches focus on either of these two sets. We investigate the similarities and differences between these linear model philosophies. We focus on (1) correlation-based approaches (CCA), (2) encoding/decoding models based on dense estimation, and (3) (adaptive) encoding/decoding models based on sparse estimation. The specific focus is on sparsity-driven adaptive encoding models and comparing the methodology in state-of-the-art models found in the auditory literature. Furthermore, we outline the main signal processing pipeline for how to identify the attended sound source in a cocktail party environment from the raw electrophysiological data with all the necessary steps, complemented with the necessary MATLAB code and the relevant references for each step. Our main aim is to compare the methodology of the available methods, and provide numerical illustrations to some of them to get a feeling for their potential. A thorough performance comparison is outside the scope of this tutorial.

show abstract

“…Past studies have established the feasibility of decoding auditory attention from both invasive 5,6,10,11 and non-invasive [13][14][15] neural recordings. Despite these advancements, existing studies predominantly employ overly simplistic acoustic scenes that do not mimic the real world scenarios 6,[10][11][12][13][14]16 . Common experimental setups have been limited to stationary talkers without background noise, and primarily focus on distinguishing between two concurrent talkers.…”

Section: Introductionmentioning

confidence: 99%

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

Choudhari,

Han,

Bickel

et al. 2024

Preprint

View full text Add to dashboard Cite

Focusing on a specific conversation amidst multiple interfering talkers presents a significant challenge, especially for the hearing-impaired. Brain-controlled assistive hearing devices aim to alleviate this problem by separating complex auditory scenes into distinct speech streams and enhancing the attended speech based on the listener’s neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, we present a realistic AAD task that mirrors the dynamic nature of acoustic settings. This task involves focusing on one of two concurrent conversations, with multiple talkers taking turns and moving continuously in space with background noise. Invasive electroencephalography (iEEG) data were collected from three neurosurgical patients as they focused on one of the two moving conversations. We propose an enhanced brain-controlled assistive hearing system that combines AAD and a binaural speaker-independent speech separation model. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve auditory attention decoding accuracy. Our subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of our approach in real-world scenarios and marks a significant step towards developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences.TAKEAWAYS-Brain-controlled hearing device for scenarios with moving conversations in multi-talker settings, closely mimicking real-world listening environments-Developed a binaural speech separation model that separates speech of moving talkers while retaining their spatial locations, enhancing auditory perception and auditory attention decoding-Proposed system enhances speech intelligibility and reduces listening effort in realistic acoustic scenes

show abstract

Comparison of Two-Talker Attention Decoding from EEG with Nonlinear Neural Networks and Linear Methods

Cited by 3 publications

References 34 publications

Robust decoding of the speech envelope from EEG recordings through deep neural networks

Robust decoding of the speech envelope from EEG recordings through deep neural networks

A Tutorial on Auditory Attention Identification Methods

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

Contact Info

Product

Resources

About