A deep learning algorithm to increase intelligibility for hearing-impaired listeners in the presence of a competing talker and reverberation

Abstract

For deep learning based speech segregation to have translational significance as a noise-reduction tool, it must perform in a wide variety of acoustic environments. In the current study, performance was examined when target speech was subjected to interference from a single talker and room reverberation. Conditions were compared in which an algorithm was trained to remove both reverberation and interfering speech, or only interfering speech. A recurrent neural network incorporating bidirectional long short-term memory was trained to estimate the ideal ratio mask corresponding to target speech. Substantial intelligibility improvements were found for hearing-impaired (HI) and normalhearing (NH) listeners across a range of target-to-interferer ratios (TIRs). HI listeners performed better with reverberation removed, whereas NH listeners demonstrated no difference. Algorithm benefit averaged 56 percentage points for the HI listeners at the least-favorable TIR, allowing these listeners to perform numerically better than young NH listeners without processing. The current study highlights the difficulty associated with perceiving speech in reverberant-noisy environments, and it extends the range of environments in which deep learning based speech segregation can be effectively applied. This increasingly wide array of environments includes not only a variety of background noises and interfering speech, but also room reverberation.

Adoption of Human Personality Development Theory Combined With Deep Neural Network in Entrepreneurship Education of College Students

“…In recent years, deep learning has been widely applied in the fields of computer vision and speech recognition, and has achieved great success ( Varadarajan et al, 2020 ). The deep neural network technology can also be used in the recommendation system, such as the music recommendation function based on the convolutional neural network, which can extract the tonal features of music; the personalized recommendation technology based on the reverse artificial neural network can recode the feature into the low-dimensional vector and calculate the prediction score using the implicit feature; the recommendation system based on the restricted Boltzmann machine can use the visual layer to automatically decode and encode the new scoring data of scored items, and predict the scores of unscored items ( Healy et al, 2019 ; Raissi et al, 2019 ; Tuli et al, 2020 ). The application of deep neural network greatly improves the computational accuracy of the current recommendation system compared with the past.…”

Using recurrent neural networks to improve the perception of speech in non-stationary noise by people with cochlear implants

“…These have been shown to improve the intelligibility of speech in stationary noise for CI users (Loizou et al, 2005;Dawson et al, 2011;Mauger et al, 2012) and NH listeners using CI simulations (Bolner et al, 2016;Lai et al, 2018). Data-based algorithms using machine-learning (ML) techniques, such as deep neural networks (DNNs) or Gaussian mixture models (GMMs), were successful for speech in non-stationary, multi-talker babble and achieved significant SI improvements for NH (Kim et al, 2009;Bentsen et al, 2018), hearing-impaired (HI; Healy et al, 2013;Healy et al, 2015;Healy et al, 2019;Chen et al, 2016;Monaghan et al, 2017;Bramsløw et al, 2018), and CI listeners (Hu and Loizou, 2010;Goehring et al, 2017;Lai et al, 2018). Improvements of more recent approaches over earlier ones have been mainly driven by two factors: the use of more powerful DNN-based regression systems instead of classification systems, and the use of a ratio mask instead of a binary mask as the training target (Madhu et al, 2013;Bentsen et al, 2018).…”

“…Improvements of more recent approaches over earlier ones have been mainly driven by two factors: the use of more powerful DNN-based regression systems instead of classification systems, and the use of a ratio mask instead of a binary mask as the training target (Madhu et al, 2013;Bentsen et al, 2018). However, all of these algorithms made use of some a priori information about the target speech and/or interfering noise by using the same target speaker (Lai et al, 2018;Chen et al, 2016), background noise (Goehring et al, 2017), or both (Kim et al, 2009;Hu and Loizou, 2010;Healy et al, 2013;Healy et al, 2015;Healy et al, 2019;Goehring et al, 2017;Lai et al, 2018;Bramsløw et al, 2018;Bentsen et al, 2018) for the training and testing of the algorithm.…”

“…RNN-LSTM algorithms have shown improved generalization using objective measures, but have not been evaluated in listening studies with CI users. However, similar types of LSTM-RNNs have recently been shown to provide benefits for speech-in-noise perception for HI listeners (Bramsløw et al, 2018;Keshavarzi et al, 2018;Keshavarzi et al, 2019;Healy et al, 2019), and they represent a promising way for improving performance for CI users in conditions with non-stationary noise that was not included in the training data.…”

“…In addition to the requirement for generalization to unseen conditions, a constraint for the practical use of MLbased algorithms in CI devices is a processing delay below about 10-20 ms, to avoid subjective disturbance during speech production and limit audio-visual asynchrony (Stone and Moore, 1999;Goehring et al, 2018;Bramsløw et al, 2018). Most of the studies described above used non-causal signal processing by providing future frames to the input of the neural network (for example, Healy et al, 2013;Healy et al, 2015;Healy et al, 2019;Chen et al, 2016). This could not be done in a hearing device due to the excessive delay it would introduce.…”