Features for Masking-Based Monaural Speech Separation in Reverberant Conditions

Delfarah, Masood; Wang, DeLiang

doi:10.1109/taslp.2017.2687829

Cited by 63 publications

(32 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Same-gender separation. Many previous speech separation methods show a drop in performance when attempting to separate speech mixtures containing same-gender speech [Delfarah and Wang 2017;. Table 4 shows a breakdown of our separation quality by the different gender combinations.…”

Section: Quantitative Analysis On Synthetic Mixturesmentioning

confidence: 99%

Looking to listen at the cocktail party

et al. 2018

View full text Add to dashboard Cite

Fig. 1. We present a model for isolating and enhancing the speech of desired speakers in a video. (a) The input is a video (frames + audio track) with one or more people speaking, where the speech of interest is interfered by other speakers and/or background noise. (b) Both audio and visual features are extracted and fed into a joint audio-visual speech separation model. The output is a decomposition of the input audio track into clean speech tracks, one for each person detected in the video (c). This allows us to then compose videos where speech of specific people is enhanced while all other sound is suppressed. Our model was trained using thousands of hours of video segments from our new dataset, AVSpeech. The "Stand-Up" video (a) is courtesy of Team Coco.We present a joint audio-visual model for isolating a single speech signal from a mixture of sounds such as other speakers and background noise. Solving this task using only audio as input is extremely challenging and does not provide an association of the separated speech signals with speakers in the video. In this paper, we present a deep network-based model that incorporates both visual and auditory signals to solve this task. The visual features are used to "focus" the audio on desired speakers in a scene and to improve the speech separation quality. To train our joint audio-visual model, we introduce AVSpeech, a new dataset comprised of thousands of hours of video segments from the Web. We demonstrate the applicability of our method to classic speech separation tasks, as well as real-world scenarios involving heated interviews, noisy bars, and screaming children, only requiring the user to specify the face of the person in the video whose speech they want to isolate. Our method shows clear advantage over stateof-the-art audio-only speech separation in cases of mixed speech. In addition, our model, which is speaker-independent (trained once, applicable to any speaker), produces better results than recent audio-visual speech separation Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

show abstract

Section: Quantitative Analysis On Synthetic Mixturesmentioning

confidence: 99%

Looking to listen at the cocktail party

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Wang et al evaluated and compared the performance of various mapping-based and masking-based targets [32]. It may be controversial to argue which method is better, yet many cases have shown that the masking-based methods (e.g., ideal ratio masks) tend to perform better than the mapping-based methods [21,32,33] in terms of enhancement results. In this work, we design the proposed model within a masking-based framework.…”

Section: Masking-based Speech Enhancementmentioning

confidence: 99%

Disentangled Feature Learning for Noise-Invariant Speech Enhancement

2019

View full text Add to dashboard Cite

Most of the recently proposed deep learning-based speech enhancement techniques have focused on designing the neural network architectures as a black box. However, it is often beneficial to understand what kinds of hidden representations the model has learned. Since the real-world speech data are drawn from a generative process involving multiple entangled factors, disentangling the speech factor can encourage the trained model to result in better performance for speech enhancement. With the recent success in learning disentangled representation using neural networks, we explore a framework for disentangling speech and noise, which has not been exploited in the conventional speech enhancement algorithms. In this work, we propose a novel noise-invariant speech enhancement method which manipulates the latent features to distinguish between the speech and noise features in the intermediate layers using adversarial training scheme. To compare the performance of the proposed method with other conventional algorithms, we conducted experiments in both the matched and mismatched noise conditions using TIMIT and TSPspeech datasets. Experimental results show that our model successfully disentangles the speech and noise latent features. Consequently, the proposed model not only achieves better enhancement performance but also offers more robust noise-invariant property than the conventional speech enhancement techniques.

show abstract

“…Since only the mixed signal y(n) is observed, the goal is to estimatexs[n] that is close to xs[n]. This problem is recently formulated as a supervised learning task, which estimates a filter (i.e., mask) for each speaker with the supervised information of ideal binary mask or ideal ratio mask [19,20,21,22]. With the introduction of deep learning techniques, the separation performance has been dramatically improved with the magnitude spectrum approximation loss [11,13,23,24].…”

Section: Monaural Speech Separation With Masksmentioning

confidence: 99%

A Shifted Delta Coefficient Objective for Monaural Speech Separation Using Multi-task Learning

Rao

Chng

et al. 2018

Interspeech 2018

View full text Add to dashboard Cite

This paper addresses the problem of monaural speech separation for simultaneous speakers. Recent studies such as uPIT, cuPIT-Grid LSTM and their variants have advanced the stateof-the-art separation models. Delta and acceleration coefficients are typically used in the objective function to capture short time dynamics. We consider that such coefficients don't benefit from the temporal information over a long range such as phoneme and syllable. In this paper, we propose a shifted delta coefficient (SDC) objective to explore the temporal information over a long range of the spectral dynamics. The SDC ensures the temporal continuity of output frames within the same speaker. In addition, we propose a novel multi-task learning framework, that we call SDC-MTL, by extending the SDC objective with a subtask of predicting the time-frequency labels ({silence, single, overlapped}) of the mixture. The experimental results show 11.7% and 3.9% relative improvements on WSJ0-2mix dataset under open conditions over the uPIT and cuPIT-Grid LSTM baselines. A further analysis shows 17.8% and 6.2% relative improvements with speakers of same gender.

show abstract

Features for Masking-Based Monaural Speech Separation in Reverberant Conditions

Cited by 63 publications

References 41 publications

Looking to listen at the cocktail party

Looking to listen at the cocktail party

Disentangled Feature Learning for Noise-Invariant Speech Enhancement

A Shifted Delta Coefficient Objective for Monaural Speech Separation Using Multi-task Learning

Contact Info

Product

Resources

About