Deep Learning Based Speech Beamforming

Qian, Kaizhi; Zhang, Yang; Chang, Shiyu; Yang, Xuesong; Florêncio, Dinei; Hasegawa‐Johnson, Mark

doi:10.1109/icassp.2018.8462430

Cited by 64 publications

(81 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other methods that are not part of these three main categories include ones that combine neural networks and beamforming in different ways. For example, [28] uses a single-channel speech enhancement network to first estimate the source of interest and then applies time-domain Wienerfiltering based beamforming.…”

Section: Introductionmentioning

confidence: 99%

FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing

Luo

Han

Mesgarani

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

111

View full text Add to dashboard Cite

Beamforming has been extensively investigated for multi-channel audio processing tasks. Recently, learning-based beamforming methods, sometimes called neural beamformers, have achieved significant improvements in both signal quality (e.g. signal-to-noise ratio (SNR)) and speech recognition (e.g. word error rate (WER)). Such systems are generally non-causal and require a large context for robust estimation of inter-channel features, which is impractical in applications requiring low-latency responses. In this paper, we propose filter-and-sum network (FaSNet), a time-domain, filterbased beamforming approach suitable for low-latency scenarios. FaSNet has a two-stage system design that first learns frame-level time-domain adaptive beamforming filters for a selected reference channel, and then calculate the filters for all remaining channels. The filtered outputs at all channels are summed to generate the final output. Experiments show that despite its small model size, FaSNet is able to outperform several traditional oracle beamformers with respect to scale-invariant signal-to-noise ratio (SI-SNR) in reverberant speech enhancement and separation tasks. Moreover, when trained with a frequency-domain objective function on the CHiME-3 dataset, FaSNet achieves 14.3% relative word error rate reduction (RWERR) compared with the baseline model. These results show the efficacy of FaSNet particularly in reverberant and noisy signal conditions.

show abstract

Section: Introductionmentioning

confidence: 99%

FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing

Luo

Han

Mesgarani

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

111

View full text Add to dashboard Cite

show abstract

“…Based on the observation in Table 1, we only report the SNR-A2T results with α = 0.3, 1, 3, as they all achieve on par or better separation performance than the standard SNR while have a much lower distortion The definition of direct-path signal can vary in different literatures. Defining the direct-path RIR filter as ±6 ms of the first peak in the RIR filter is the same as [42], however the range can be relaxed to cover more early reverberation components similar to [21]. Here we also investigate the effect of different definitions of direct-path signals.…”

Section: Resultsmentioning

confidence: 99%

Distortion-controlled Training for End-to-end Reverberant Speech Separation with Auxiliary Autoencoding Loss

Luo

Han

Mesgarani

2020

Preprint

View full text Add to dashboard Cite

The performance of speech enhancement and separation systems in anechoic environments has been significantly advanced with the recent progress in end-to-end neural network architectures. However, the performance of such systems in reverberant environments is yet to be explored. A core problem in reverberant speech separation is about the training and evaluation metrics. Standard timedomain metrics may introduce unexpected distortions during training and fail to properly evaluate the separation performance due to the presence of the reverberations. In this paper, we first introduce the "equal-valued contour" problem in reverberant separation where multiple outputs can lead to the same performance measured by the common metrics. We then investigate how "better" outputs with lower target-specific distortions can be selected by auxiliary autoencoding training (A2T). A2T assumes that the separation is done by a linear operation on the mixture signal, and it adds an loss term on the autoencoding of the direct-path target signals to ensure that the distortion introduced on the direct-path signals is controlled during separation. Evaluations on separation signal quality and speech recognition accuracy show that A2T is able to control the distortion on the direct-path signals and improve the recognition accuracy.

show abstract

“…In recent years, research on DNNs has become more popular and DNN-based source separation methods have been proposed [17][18][19][20][21]. IDLMA [17] has been proposed for determined audio source separation.…”

Section: A Motivation and Frameworkmentioning

confidence: 99%

Speech Enhancement by Noise Self-Supervised Rank-Constrained Spatial Covariance Matrix Estimation via Independent Deeply Learned Matrix Analysis

Misawa,

Takamune,

Nakamura

et al. 2021

Preprint

View full text Add to dashboard Cite

Rank-constrained spatial covariance matrix estimation (RCSCME) is a method for the situation that the directional target speech and the diffuse noise are mixed. In conventional RCSCME, independent low-rank matrix analysis (ILRMA) is used as the preprocessing method. We propose RCSCME using independent deeply learned matrix analysis (IDLMA), which is a supervised extension of ILRMA. In this method, IDLMA requires deep neural networks (DNNs) to separate the target speech and the noise. We use Denoiser, which is a single-channel speech enhancement DNN, in IDLMA to estimate not only the target speech but also the noise. We also propose noise self-supervised RCSCME, in which we estimate the noise-only time intervals using the output of Denoiser and design the prior distribution of the noise spatial covariance matrix for RCSCME. We confirm that the proposed methods outperform the conventional methods under several noise conditions.

show abstract

Deep Learning Based Speech Beamforming

Cited by 64 publications

References 24 publications

FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing

FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing

Distortion-controlled Training for End-to-end Reverberant Speech Separation with Auxiliary Autoencoding Loss

Speech Enhancement by Noise Self-Supervised Rank-Constrained Spatial Covariance Matrix Estimation via Independent Deeply Learned Matrix Analysis

Contact Info

Product

Resources

About