High-Fidelity Audio Generation and Representation Learning With Guided Adversarial Autoencoder

Abstract

Generating high-fidelity conditional audio samples and learning representation from unlabelled audio data are two challenging problems in machine learning research. Recent advances in the Generative Adversarial Neural Networks (GAN) architectures show great promise in addressing these challenges. To learn powerful representation using GAN architecture, it requires superior sample generation quality, which requires an enormous amount of labelled data. In this paper, we address this issue by proposing Guided Adversarial Autoencoder (GAAE), which can generate superior conditional audio samples from unlabelled audio data using a small percentage of labelled data as guidance. Representation learned from unlabelled data without any supervision does not guarantee its' usability for any downstream task. On the other hand, during the representation learning, if the model is highly biased towards the downstream task, it losses its generalisation capability. This makes the learned representation hardly useful for any other tasks that are not related to that downstream task. The proposed GAAE model also address these issues. Using this superior conditional generation, GAAE can learn representation specific to the downstream task. Furthermore, GAAE learns another type of representation capturing the general attributes of the data, which is independent of the downstream task at hand. Experimental results involving the S09 and the NSynth dataset attest the superior performance of GAAE compared to the state-of-the-art alternatives.

PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification

“…(3) Synthesis and Classification Evaluation Metrics: For the speech synthesis task, the Frechet Inception Distance (FID) [38] is selected to evaluate the sample quality that computes the Frechet Distance [39] between two multivariate Gaussian distributions for the synthetic and real samples. We follow a standard FID setup in [15] to evaluate the quality of over 10,000 synthetic speech samples generated from random noise. For the speech command classification task, classification accuracy is used to evaluate the student model.…”

“…PATE-GAN [13] tries to overcome this issue by incorporating a generative block jointly trained with the PATE block; the goal is providing enough synthetic data to train deep models effectively. Unfortunately, PATE-GAN does not work well for high dimensional data synthesis (e.g., images), as demonstrated in recent studies [14,15]. Moreover, generating speech samples is a challenging task, as shown in recent studies about neural vocoders [16,17].…”

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

“…In our previous work [51], we first proposed an IoT System of Systems framework of audio generation for visual inputs exploiting BigGAN [14]. Recently, authors in [52] utilized BigGAN architecture for adversarial audio generation in a guided manner. Our proposed FoleyGAN architecture is a novel approach to apply BigGAN in the movie sound production domain where we are synthesizing the audio for silent movie clips using visual and temporal guidance.…”

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

“…In our previous work [43] for the first we propose a System of Systems framework of audio generation for visual inputs exploiting BigGAN [14]. Recently, authors in [44] utilizes BigGAN architecture for adversarial audio generation in guided manner. Our proposed FoleyGAN architecture is a noble approach to apply BigGAN in movie sound production domain where we are synthesizing the audio for silent movie clips taking visual guidance.…”