DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Abstract

Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e.g., mel-spectrogram) given a music score. Previous singing acoustic models adopt a simple loss (e.g., L1 and L2) or generative adversarial network (GAN) to reconstruct the acoustic features, while they suffer from over-smoothing and unstable training issues respectively, which hinder the naturalness of synthesized singing. In this work, we propose DiffSinger, an acoustic model for SVS based on the diffusion probabilistic model. DiffSinger is a parameterized Markov chain that iteratively converts the noise into mel-spectrogram conditioned on the music score. By implicitly optimizing variational bound, DiffSinger can be stably trained and generate realistic outputs. To further improve the voice quality and speed up inference, we introduce a shallow diffusion mechanism to make better use of the prior knowledge learned by the simple loss. Specifically, DiffSinger starts generation at a shallow step smaller than the total number of diffusion steps, according to the intersection of the diffusion trajectories of the ground-truth mel-spectrogram and the one predicted by a simple mel-spectrogram decoder. Besides, we propose boundary prediction methods to locate the intersection and determine the shallow step adaptively. The evaluations conducted on a Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work. Extensional experiments also prove the generalization of our methods on text-to-speech task (DiffSpeech). Audio samples: https://diffsinger.github.io. Codes: https://github.com/MoonInTheRiver/DiffSinger.

DiffSDS: A language diffusion model for protein backbone inpainting under geometric conditions and constraints

“…Diffusion models (Sohl-Dickstein et al, 2015;Ho et al, 2020;Cao et al, 2022) are a class of generative models that have achieved impressive results in image (Song et al, 2020;Lugmayr et al, 2022;Whang et al, 2022;Baranchuk et al, 2021;Wolleb et al, 2022), speech (Lee & Han, 2021;Chen et al, 2020;Kong et al, 2020;Liu et al, 2022a) and text (Li et al, 2022;Chen et al, 2022;Austin et al, 2021) synthesis. Recently, FoldingDiff (Wu et al, 2022a) shows that language models could be used for unconditional protein generation.…”

Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics

“…To verify the singability in the end-to-end manner, we additionally use an open-source Chinese singing voice synthesis (SVS) model (Liu et al, 2022a) to supply the annotators with an actual audio rendition of the songs for more intuitive feeling. We randomly select 20 verses from the test set and show the music sheets and synthesized singing voice (see Appendix E) of each translated verse to five annotators.…”

DiffSTG: Probabilistic Spatio-Temporal Graph Forecasting with Denoising Diffusion Models

“…Denoising network θ in previous works can be mainly classified into two classes, Unet-based architecture (Ronneberger et al, 2015) for image-related tasks (Rombach et al, 2022;Voleti et al, 2022;Ho et al, 2020), andWaveNet-based architecture (van den Oord et al, 2016) for sequence-related tasks (Kong et al, 2020;Liu et al, 2022b;Kim et al, 2020). These networks consider the input as either grids or segments, lacking the ability to capture spatiotemporal correlations in STG data.…”