Unsupervised Audio Source Separation Using Generative Priors

Abstract

State-of-the-art under-determined audio source separation systems rely on supervised end-end training of carefully tailored neural network architectures operating either in the time or the spectral domain. However, these methods are severely challenged in terms of requiring access to expensive source level labeled data and being specific to a given set of sources and the mixing process, which demands complete re-training when those assumptions change. This strongly emphasizes the need for unsupervised methods that can leverage the recent advances in data-driven modeling, and compensate for the lack of labeled data through meaningful priors. To this end, we propose a novel approach for audio source separation based on generative priors trained on individual sources. Through the use of projected gradient descent optimization, our approach simultaneously searches in the source-specific latent spaces to effectively recover the constituent sources. Though the generative priors can be defined in the time domain directly, e.g. WaveGAN, we find that using spectral domain loss functions for our optimization leads to good-quality source estimates. Our empirical studies on standard spoken digit and instrument datasets clearly demonstrate the effectiveness of our approach over classical as well as state-of-the-art unsupervised baselines.

BioCPPNet: Automatic Bioacoustic Source Separation with Deep Neural Networks

“…Mixture invariant training (MixIT) 53 represents an entirely unsupervised approach and requires only single-channel acoustic mixtures, though this technique may be limited in the bioacoustic domain as it necessitates large quantities of well-defined mixtures. Another unsupervised technique includes a Bayesian approach employing deep generative priors [54][55][56] , but this method may be limited to data within close bounds of the training sets since the distributions learned by acoustic deep generative models may not exhibit the right properties for probabilistic source separation 57 . We also suggest self-supervised pre-training 30,58,59 on relevant proxy tasks to enhance performance, especially in the low-data bioacoustic domain.…”

Differentiable Digital Signal Processing Mixture Model for Synthesis Parameter Extraction from Mixture of Harmonic Sounds

“…Although the DNN-based methods show superior performance in the usual supervised source separation setting [9], in the score-informed setting, this NMF-based method works better than DNN-based methods [15]. The audio source separation method presented in [21] uses pretrained instrument sound synthesizers based on generative adversar-ial networks (GANs). The GANs convert random vectors into audio signals.…”

Blind Source Separation in Polyphonic Music Recordings Using Deep Neural Networks Trained via Policy Gradients

“…Given this success, it was natural to also apply this method to audio data, leading to the deep audio prior approach by Tian et al [28]. Based on this, Narayanaswamy et al [29] used generative adversarial networks (GANs) trained on unlabeled training data as priors, further improving the quality of the output signals.…”