HiFi-GAN-2: Studio-Quality Speech Enhancement via Generative Adversarial Networks Conditioned on Acoustic Features

Su, Jiaqi; Jin, Zeyu; Finkelstein, Adam

doi:10.1109/waspaa52581.2021.9632770

Cited by 35 publications

(11 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years GANs have been successfully applied to a wide range of normal speech processing tasks including, but not limited to: 1) speech synthesis [74], [75]; 2) voice conversion [76], [77]; 3) speech enhancement [78], [79]; 4) code-switching sentence generation [80], [81]; 5) speech emotion recognition [82], [83]; 6) speaker verification [84]- [86] and 7) speech recognition [87]- [89].…”

Section: Corpusmentioning

confidence: 99%

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Jin¹,

Deng²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. It is difficult to collect large quantities of such data for ASR system development due to the mobility issues often found among these users. To this end, data augmentation techniques play a vital role. In contrast to existing data augmentation techniques only modifying the speaking rate or overall shape of spectral contour, fine-grained spectro-temporal differences between dysarthric, elderly and normal speech are modelled using a novel set of speaker dependent (SD) generative adversarial networks (GAN) based data augmentation approaches in this paper. These flexibly allow both: a) temporal or speed perturbed normal speech spectra to be modified and closer to those of an impaired speaker when parallel speech data is available; and b) for non-parallel data, the SVD decomposed normal speech spectral basis features to be transformed into those of a target elderly speaker before being re-composed with the temporal bases to produce the augmented data for state-of-the-art TDNN and Conformer ASR system training. Experiments are conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed GAN based data augmentation approaches consistently outperform the baseline speed perturbation method by up to 0.91% and 3.0% absolute (9.61% and 6.4% relative) WER reduction on the TORGO and DementiaBank data respectively. Consistent performance improvements are retained after applying LHUC based speaker adaptation.

show abstract

Section: Corpusmentioning

confidence: 99%

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Jin¹,

Deng²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Methods that operate on the time-frequency domain generally produce audible artifacts due to the use of phase reconstruction algorithms like the Griffin-Lim algorithm [14]. A recent work addresses this with neural-network based vocoders [15], yet its quality is not on par with an end-to-end approach [16]. Alternatively, methods that work on the time domain typically require more training steps [1].…”

Section: Related Workmentioning

confidence: 99%

Music Enhancement via Image Translation and Vocoding

Kandpal¹,

Nieto²,

Jin³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Consumer-grade music recordings such as those captured by mobile devices typically contain distortions in the form of background noise, reverb, and microphone-induced EQ. This paper presents a deep learning approach to enhance low-quality music recordings by combining (i) an image-to-image translation model for manipulating audio in its mel-spectrogram representation and (ii) a music vocoding model for mapping synthetically generated mel-spectrograms to perceptually realistic waveforms. We find that this approach to music enhancement outperforms baselines which use classical methods for mel-spectrogram inversion and an end-to-end approach directly mapping noisy waveforms to clean waveforms. Additionally, in evaluating the proposed method with a listening test, we analyze the reliability of common audio enhancement evaluation metrics when used in the music domain.

show abstract

“…The latter is particularly fruitful, as it has been shown to be effective while operating several times faster than real-time. Each layer in P can optionally be locally conditioned on acoustic features as in [16]. This allows the filtering operation to adapt itself as a function of a more global context vector.…”

Section: Extensions Using Black-box Residual Postnetsmentioning

confidence: 99%

“…Meanwhile, adversarial training promotes plausible system outputs as determined by a discriminator network [7]. Hybrid approaches combining these methodologies have grown increasingly more common [10,17,16], and balance average system performance with output signal plausibility. To this end, we adopt the adversarial loss and multi-scale discriminator architecture in [7].…”

Section: Adversarial Trainingmentioning

confidence: 99%

Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfer

Nercessian¹

2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a differentiable WORLD synthesizer and demonstrate its use in end-to-end audio style transfer tasks such as (singing) voice conversion and the DDSP timbre transfer task. Accordingly, our baseline differentiable synthesizer has no model parameters, yet it yields adequate synthesis quality. We can extend the baseline synthesizer by appending lightweight black-box postnets which apply further processing to the baseline output in order to improve fidelity. An alternative differentiable approach considers extraction of the source excitation spectrum directly, which can improve naturalness albeit for a narrower class of style transfer applications. The acoustic feature parameterization used by our approaches has the added benefit that it naturally disentangles pitch and timbral information so that they can be modeled separately. Moreover, as there exists a robust means of estimating these acoustic features from monophonic audio sources, it allows for parameter loss terms to be added to an end-to-end objective function, which can help convergence and/or further stabilize (adversarial) training.

show abstract

HiFi-GAN-2: Studio-Quality Speech Enhancement via Generative Adversarial Networks Conditioned on Acoustic Features

Cited by 35 publications

References 23 publications

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Music Enhancement via Image Translation and Vocoding

Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfer

Contact Info

Product

Resources

About