Speech Enhancement Aided End-To-End Multi-Task Learning for Voice Activity Detection

Tan, Xu; Zhang, Xiaolei

doi:10.1109/icassp39728.2021.9414445

Cited by 35 publications

(10 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A diffuse noise generator 2 is used to simulated uncorrelated diffuse noise. The noise source for training and development is a large-scale noise library containing over 20000 noise segments [28], and the noise source for test is the noise segments from CHiME-3 dataset [29] and NOISEX-92 corpus [30]. We randomly generate 20 channels for training and development, and 20, 30 and 40 channels respectively for test.…”

Section: Experiments a Datasetmentioning

confidence: 99%

Attention-based multi-channel speaker verification with ad-hoc microphone arrays

Liang,

Chen,

Guan

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, ad-hoc microphone array has been widely studied. Unlike traditional microphone array settings, the spatial arrangement and number of microphones of ad-hoc microphone arrays are not known in advance, which hinders the adaptation of traditional speaker verification technologies to ad-hoc microphone arrays. To overcome this weakness, in this paper, we propose attention-based multi-channel speaker verification with ad-hoc microphone arrays. Specifically, we add an interchannel processing layer and a global fusion layer after the pooling layer of a single-channel speaker verification system. The inter-channel processing layer applies a so-called residual self-attention along the channel dimension for allocating weights to different microphones. The global fusion layer integrates all channels in a way that is independent to the number of the input channels. We further replace the softmax operator in the residual self-attention with sparsemax, which forces the channel weights of very noisy channels to zero. Experimental results with ad-hoc microphone arrays of over 30 channels demonstrate the effectiveness of the proposed methods. For example, the multichannel speaker verification with sparsemax achieves an equal error rate (EER) of over 20% lower than oracle one-best system on semi-real data sets, and over 30% lower on simulation data sets, in test scenarios with both matched and mismatched channel numbers.

show abstract

Section: Experiments a Datasetmentioning

confidence: 99%

Attention-based multi-channel speaker verification with ad-hoc microphone arrays

Liang,

Chen,

Guan

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This method can be used, for example, to identify an object within an image, recognize the overall scene and generate a verbal caption for it (e.g., [17], [18]). Also, for speech processing, MTL can be used to improve speech activity detection (e.g., [19], [20]).…”

Section: Introductionmentioning

confidence: 99%

Deep Latent Fusion Layers for Binaural Speech Enhancement

Gajęcki¹,

Nogueira²

2023

Preprint

View full text Add to dashboard Cite

<p>In this work, we address the problem of speech enhancement in the context of binaural hearing. We propose deep learning models which are connected by ``fusion layers" that perform Hadamard products between specific generated latent representations. Fusion layers are inspired by multi-task learning approaches that combine and/or share weights between models that tackle related tasks. We first present a general fusion model and show that this approach is able to fit synthetic data better than independent linear models, equalize activation variance between learning modules, and exploit input data redundancy to improve the training error. We then apply the concept of fusion layers to enhance speech in binaural listening conditions. Our results show that the proposed approach improves speech enhancement performance on unseen data with respect to the independent models. However, we observe a trade-off between speech enhancement performance and predicted speech intelligibility based on a short-time objective binaural speech intelligibility index, potentially due to distortions introduced by fully fused models.</p> <p>Results also suggest that fusion layers should share parameterized latent representations in order to properly exploit the information contained in each listening side. In general, this work shows that sharing information between speech enhancement modules may be promising to improve binaural speech enhancement while keeping the number of trainable parameters constant and improving generalization. </p>

show abstract

“…Some work uses advanced speech enhancement models, like denoising variational autoencoders [25], convolutionalrecurrent-network-based speech enhancement, residual-convolutional neural-network [26] and U-Net [27], to extract denoised features for VAD. In [28,29], the works optimizes VAD and speech enhancement jointly in the framework of multitask learning. In [30], information about the speaker was exploited for the VAD, which makes VAD able to extract speaker-dependent speech segments.…”

Section: Introductionmentioning

confidence: 99%

AUC optimization for deep learning-based voice activity detection

Zhang

2022

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Voice activity detection (VAD) based on deep neural networks (DNN) have demonstrated good performance in adverse acoustic environments. Current DNN-based VAD optimizes a surrogate function, e.g., minimum cross-entropy or minimum squared error, at a given decision threshold. However, VAD usually works on-the-fly with a dynamic decision threshold, and the receiver operating characteristic (ROC) curve is a global evaluation metric for VAD at all possible decision thresholds. In this paper, we propose to maximize the area under the ROC curve (MaxAUC) by DNN, which can maximize the performance of VAD in terms of the entire ROC curve. However, the objective of the AUC maximization is nondifferentiable. To overcome this difficulty, we relax the nondifferentiable loss function to two differentiable approximation functions—sigmoid loss and hinge loss. To study the effectiveness of the proposed MaxAUC-DNN VAD, we take either a standard feedforward neural network or a bidirectional long short-term memory network as the DNN model with either the state-of-the-art multi-resolution cochleagram or short-term Fourier transform as the acoustic feature. We conducted noise-independent training to all comparison methods. Experimental results show that taking AUC as the optimization objective results in higher performance than the common objectives of the minimum squared error and minimum cross-entropy. The experimental conclusion is consistent across different DNN structures, acoustic features, noise scenarios, training sets, and languages.

show abstract

Speech Enhancement Aided End-To-End Multi-Task Learning for Voice Activity Detection

Cited by 35 publications

References 24 publications

Attention-based multi-channel speaker verification with ad-hoc microphone arrays

Attention-based multi-channel speaker verification with ad-hoc microphone arrays

Deep Latent Fusion Layers for Binaural Speech Enhancement

AUC optimization for deep learning-based voice activity detection

Contact Info

Product

Resources

About