Cognitive-Driven Binaural Beamforming Using EEG-Based Auditory Attention Decoding

Abstract

The performance of speech enhancement algorithms in a multi-speaker scenario depends on correctly identifying the target speaker to be enhanced. Auditory attention decoding (AAD) methods allow to identify the target speaker which the listener is attending to from single-trial EEG recordings. Aiming at enhancing the target speaker and suppressing interfering speakers, reverberation and ambient noise, in this paper we propose a cognitive-driven multi-microphone speech enhancement system, which combines a neural-network-based mask estimator, weighted minimum power distortionless response convolutional beamformers and AAD. To control the suppression of the interfering speaker, we also propose an extension incorporating an interference suppression constraint. The experimental results show that the proposed system outperforms the state-of-the-art cognitive-driven speech enhancement systems in challenging reverberant and noisy conditions.

EEG-based diagnostics of the auditory system using cochlear implant electrodes as sensors

“…Different speech streams are represented as separate auditory objects in the brain [Shinn-Cunningham, 2008] and the representation of the attended speaker is stronger [O'Sullivan et al, 2015]. Machine learning techniques have been successfully applied to EEG recordings to determine to which of multiple speakers the listener is attending, which can be used to steer the directional gain and noise suppression in so-called neuro-steered hearing devices [Van Eyndhoven et al, 2016;Das et al, 2018;Geirnaert et al, 2019;Aroudi and Doclo, 2020]. While most of the published research so far has focused on neuro-steered hearing aids, it has been shown that AAD in CI users is feasible [Nogueira et al, 2019;Paul et al, 2020].…”

“…These are important characteristics to consider for the future integration of EEG systems into cochlear implants, and may guide design decisions for optimal measurement of objective measures using CIs. Potential applications of EEG-CI systems are chronic neuro-monitoring of CI users, remote fitting and diagnostics, and so-called neuro-steered hearing devices [Van Eyndhoven et al, 2016;Geirnaert et al, 2019;Aroudi and Doclo, 2020], which autonomously adapt their fitting based on responses from the user's brain.…”

Linear versus deep learning methods for noisy speech separation for EEG-informed attention decoding

“…Therefore, there was a mismatch between acoustic conditions in the AAD experiments and the audio processing modules. In [23], speech enhancement was achieved using binaural direction-of-arrival (DOA) estimators with linearly constrained minimum variance (LCMV) beamformers. The AAD module used EEG signals recorded in two relatively high signal-to-noise ratio (SNR) conditions and two reverberation conditions.…”

“…The audio processing in [20,22,23] is based on linear beamforming and linear source separation techniques, which do not require any training data and have the advantage of being computationally cheap. In recent years, DNN-based approaches have become a popular alternative to solve the speaker separation problem, particularly for the challenging single-microphone scenario [24][25][26] and the even more challenging scenario with additional background noise [27,28].…”

“…To summarize: The audio processing of [20,22] was tested in various noisy conditions with different angles between speakers, but the EEG recordings were performed in a noise-free condition with 180°s eparation, resulting in a mismatch between the acoustic conditions of the microphone and neural signals. This mismatch does not occur in [23], but only mild noisy conditions are considered and the speaker positions are the same in all conditions. Finally, it is noticed that in [32] ECoG is used instead of EEG.…”