UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

Jang, Won; Lim, Dan; Yoon, Jaesam; Kim, Bong-Wan; Kim, Jun Tae

doi:10.21437/interspeech.2021-1016

Cited by 54 publications

(27 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The LeakyReLU with 𝛼 = 0.2 is set as the activation function for all layers except for the input layer. Most GAN-based vocoders are also trained with multi-scale discriminators by pre-processing the waveform into waveforms with different sample rates [16] or spectrograms with different STFT parameters [22]. However, it is difficult to do so in the training of the acoustic model, since the Mel spectrogram is hard to be down-sampled well or converted to other features with different scales.…”

Section: Model Architecturementioning

confidence: 99%

“…To address the aforementioned problem, the generative adversarial network (GAN) shows great potential, which has been widely applied in speech synthesis, including statistical speech synthesis [12,13], spectrogram post-filter [14], spectrogram super-resolution [15], and neural vocoder [16,17,18]. In this framework, NAR-TTS can be enhanced by only using a discriminator.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Multi-Scale Time-Frequency Spectrogram Discriminator for GAN-based Non-Autoregressive TTS

Guo¹,

Liu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The generative adversarial network (GAN) has shown its outstanding capability in improving Non-Autoregressive TTS (NAR-TTS) by adversarially training it with an extra model that discriminates between the real and the generated speech. To maximize the benefits of GAN, it is crucial to find a powerful discriminator that can capture rich distinguishable information. In this paper, we propose a multi-scale time-frequency spectrogram discriminator to help NAR-TTS generate high-fidelity Mel-spectrograms. It treats the spectrogram as a 2D image to exploit the correlation among different components in the time-frequency domain. And a U-Net-based model structure is employed to discriminate at different scales to capture both coarse-grained and fine-grained information. We conduct subjective tests to evaluate the proposed approach. Both multi-scale and time-frequency discriminating bring significant improvement in the naturalness and fidelity. When combining the neural vocoder, it is shown more effective and concise than fine-tuning the vocoder. Finally, we visualize the discriminating maps to compare their difference to verify the effectiveness of multiscale discriminating.

show abstract

Section: Model Architecturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Multi-Scale Time-Frequency Spectrogram Discriminator for GAN-based Non-Autoregressive TTS

Guo¹,

Liu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Mel-spectrograms were obtained by applying an 80 band mel filter bank. We adopted Univnet-c16 [17] as a vocoder, which has beneficial lightweight properties by using a locationvariable convolution (LVC) technique [18]. The dimension of all hidden embeddings was set to 256, and the receptive field of the vocoder, auxiliary predictor, and the conditional discriminator was set to 19.…”

Section: Experiments 41 Experimental Setupmentioning

confidence: 99%

AILTTS: Adversarial Learning of Intermediate Acoustic Feature for End-to-End Lightweight Text-to-Speech

Yoon¹,

Um²,

Kim³

et al. 2022

Preprint

View full text Add to dashboard Cite

The quality of end-to-end neural text-to-speech (TTS) systems highly depends on the reliable estimation of intermediate acoustic features from text inputs. To reduce the complexity of the speech generation process, several non-autoregressive TTS systems directly find a mapping relationship between text and waveforms. However, the generation quality of these system is unsatisfactory due to the difficulty in modeling the dynamic nature of prosodic information. In this paper, we propose an effective prosody predictor that successfully replicates the characteristics of prosodic features extracted from mel-spectrograms. Specifically, we introduce a generative model-based conditional discriminator to enable the estimated embeddings have highly informative prosodic features, which significantly enhances the expressiveness of generated speech. Since the estimated embeddings obtained by the proposed method are highly correlated with acoustic features, the time-alignment of input texts and intermediate features is greatly simplified, which results in faster convergence. Our proposed model outperforms several publicly available models based on various objective and subjective evaluation metrics, even using a relatively small number of parameters.

show abstract

“…An acoustic feature generator can be autoregressive and attention-based for implicit speech-text alignments [1], [2] or it can be non-autoregressive for efficient parallel inference and duration informed for robustness on synthesis error [3], [4], [5]. There is lots of research on neural vocoder as well and some of the famous, widely used include [6], [7], normalizing flow based [8] and generative adversarial network (GAN) based [9], [10], [11], [12].…”

Section: Introductionmentioning

confidence: 99%

“…Note that a neural vocoder takes a ground-truth acoustic feature for training and a predicted one from an acoustic feature generator for inference. For optimal performance, we can further train a pre-trained neural vocoder with predicted acoustic features, which is called fine-tuning [12], [13]. Or we can train a neural vocoder with predicted acoustic feature from the beginning [1].…”

Section: Introductionmentioning

confidence: 99%

JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech

Lim¹,

Jung²,

Kim³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In neural text-to-speech (TTS), two-stage system or a cascade of separately learned models have shown synthesis quality close to human speech. For example, FastSpeech2 transforms an input text to a mel-spectrogram and then HiFi-GAN generates a raw waveform from a mel-spectogram where they are called an acoustic feature generator and a neural vocoder respectively. However, their training pipeline is somewhat cumbersome in that it requires a fine-tuning and an accurate speech-text alignment for optimal performance. In this work, we present endto-end text-to-speech (E2E-TTS) model which has a simplified training pipeline and outperforms a cascade of separately learned models. Specifically, our proposed model is jointly trained FastSpeech2 and HiFi-GAN with an alignment module. Since there is no acoustic feature mismatch between training and inference, it does not requires fine-tuning. Furthermore, we remove dependency on an external speech-text alignment tool by adopting an alignment learning objective in our joint training framework. Experiments on LJSpeech corpus shows that the proposed model outperforms publicly available, stateof-the-art implementations of ESPNet2-TTS on subjective evaluation (MOS) and some objective evaluations.

show abstract

UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

Cited by 54 publications

References 0 publications

A Multi-Scale Time-Frequency Spectrogram Discriminator for GAN-based Non-Autoregressive TTS

A Multi-Scale Time-Frequency Spectrogram Discriminator for GAN-based Non-Autoregressive TTS

AILTTS: Adversarial Learning of Intermediate Acoustic Feature for End-to-End Lightweight Text-to-Speech

JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech

Contact Info

Product

Resources

About