VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking

Muckenhirn, Hannah; Moreno, Ignacio López; Hershey, John R.; Wilson, Kevin; Sridhar, Prashant; Wang, Quan; Saurous, Rif A.; Weiss, Ron; Jia, Yuheng; Wu, Zelin

doi:10.21437/interspeech.2019-1101

Cited by 267 publications

(243 citation statements)

References 20 publications

Supporting

Mentioning

240

Contrasting

Order By: Relevance

“…Another approach to the permutation indeterminacy problem is "informed extraction", which makes use of additional information to distinguish a target speaker from other participating speakers. Use of visual information [9,10], audio snippets of the speakers [11,12,13], or their locations [14,15,16] were investigated. On top of, or aside from, these algorithmic improvements, researchers also sought more effective input features [17,18] and model architectures [19,20,21].…”

Section: Introductionmentioning

confidence: 99%

Continuous Speech Separation: Dataset and Analysis

Chen

Yoshioka

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

177

163

View full text Add to dashboard Cite

This paper describes a dataset and protocols for evaluating continuous speech separation algorithms. Most prior studies on speech separation use pre-segmented signals of artificially mixed speech utterances which are mostly fully overlapped, and the algorithms are evaluated based on signal-to-distortion ratio or similar performance metrics. However, in natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. In addition, the signal-based metrics have very weak correlations with automatic speech recognition (ASR) accuracy. We think that not only does this make it hard to assess the practical relevance of the tested algorithms, it also hinders researchers from developing systems that can be readily applied to real scenarios. In this paper, we define continuous speech separation (CSS) as a task of generating a set of non-overlapped speech signals from a continuous audio stream that contains multiple utterances that are partially overlapped by a varying degree. A new real recorded dataset, called LibriCSS, is derived from LibriSpeech by concatenating the corpus utterances to simulate a conversation and capturing the audio replays with far-field microphones. A Kaldi-based ASR evaluation protocol is also established by using a well-trained multi-conditional acoustic model. By using this dataset, several aspects of a recently proposed speakerindependent CSS algorithm are investigated. The dataset and evaluation scripts are available to facilitate the research in this direction.

show abstract

Section: Introductionmentioning

confidence: 99%

Continuous Speech Separation: Dataset and Analysis

Chen

Yoshioka

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

177

163

View full text Add to dashboard Cite

show abstract

“…A related research named VoiceFilter [11] (VF) was published recently, targeting speaker-dependent speech separation. VF uses target speaker embedding extracted from a pretrained and fixed speaker recognition network to separate the target voice among multiple speakers.…”

Section: Mse Lossmentioning

confidence: 99%

“…For a reliable comparison between the conventional VF and our proposed method, we use VCTK dataset [16] as indicated in [11], to train and evaluate our network. Moreover, 99 and 10 speakers are randomly selected for each training and validation, followed by the data generation workflow also referred in [11]. Parameter settings are mostly the same as the conventional VF, except for the d-vector dimension being set to 512.…”

Section: Data Generation and Settingsmentioning

confidence: 99%

The Sound of My Voice: Speaker Representation Loss for Target Voice Separation

Mun

Choe

Huh

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Research on content and style representations has been widely studied in the field of style transfer. In this paper, we propose a new loss function using speaker content representation for audio source separation, and we call it a speaker representation loss (SRL). The objective is to extract the sound of my voice from the noisy input and also remove it from the residual components. Compared to the conventional spectral reconstruction, our proposed framework maximizes the use of target speaker information by minimizing the distance between the content of target speaker and source separation output. We also propose triplet SRL as an additional criterion to remove the target speaker information from residual spectrogram output. VoiceFilter framework is adopted to evaluate source separation performance using the VCTK database, and we achieved improved performances compared to the baseline loss function without any additional network parameters.

show abstract

“…A fundamental problem in machine hearing is deconstructing an acoustic mixture into its constituent sounds. This has been done quite successfully for certain classes of sounds, such as separating speech from nonspeech interference [1,2] or speech from speech [3,4,5,6,7]. However, the more general problem of separating arbitrary classes of sound, known as "universal sound separation", has only recently been addressed [8].…”

Section: Introductionmentioning

confidence: 99%

“…In [16], a conditional variational autoencoder is proposed in order to generate prosodic features for speech synthesis by sampling prosodic embeddings from the bottleneck representation. Specifically for separation tasks, speaker-discriminative embeddings are produced for targeted voice separation in [6] and for diarization in [17] yielding a significant improvement over the unconditional separation framework. Recent works [18,19] have utilized conditional embeddings for each music class in order to boost the performance of a deep attractor-network [20] for music separation.…”

Section: Introductionmentioning

confidence: 99%

Improving Universal Sound Separation Using Sound Classification

Tzinis

Wisdom

Hershey

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Deep learning approaches have recently achieved impressive performance on both audio source separation and sound classification. Most audio source separation approaches focus only on separating sources belonging to a restricted domain of source classes, such as speech and music. However, recent work has demonstrated the possibility of "universal sound separation", which aims to separate acoustic sources from an open domain, regardless of their class. In this paper, we utilize the semantic information learned by sound classifier networks trained on a vast amount of diverse sounds to improve universal sound separation. In particular, we show that semantic embeddings extracted from a sound classifier can be used to condition a separation network, providing it with useful additional information. This approach is especially useful in an iterative setup, where source estimates from an initial separation stage and their corresponding classifier-derived embeddings are fed to a second separation network. By performing a thorough hyperparameter search consisting of over a thousand experiments, we find that classifier embeddings from clean sources provide nearly one dB of SNR gain, and our best iterative models achieve a significant fraction of this oracle performance, establishing a new state-of-the-art for universal sound separation.

show abstract

VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking

Cited by 267 publications

References 20 publications

Continuous Speech Separation: Dataset and Analysis

Continuous Speech Separation: Dataset and Analysis

The Sound of My Voice: Speaker Representation Loss for Target Voice Separation

Improving Universal Sound Separation Using Sound Classification

Contact Info

Product

Resources

About