VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis

Cong, Jian; Xue, Heyang; Xie, Lei; Zhu, Pengcheng; Bi, Mengxiao

doi:10.1109/icassp43922.2022.9747664

Cited by 33 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• FastSpeech2 (Ren et al 2020). Although it is a speech synthesis model, employing it for singing voice synthesis tasks can achieve good results (Zhuang et al 2021;Zhang et al 2022b). We use the same parameter settings as FT-GAN, with the encoder and the decoder being 4-layer 8-head Feed-Forward Transformers with a hidden size of 256.…”

Section: Experiments Experiments Settingsmentioning

confidence: 99%

See 1 more Smart Citation

FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis

Zheng,

Bai,

Shi

et al. 2024

AAAI

View full text Add to dashboard Cite

Although singing voice synthesis (SVS) has made significant progress recently, with its unique styles and various genres, Chinese opera synthesis requires greater attention but is rarely studied for lack of training data and high expressiveness. In this work, we build a high-quality Gezi Opera (a type of Chinese opera popular in Fujian and Taiwan) audio-text alignment dataset and formulate specific data annotation methods applicable to Chinese operas. We propose FT-GAN, an acoustic model for fine-grained tune modeling in Chinese opera synthesis based on the empirical analysis of the differences between Chinese operas and pop songs. To further improve the quality of the synthesized opera, we propose a speech pre-training strategy for additional knowledge injection. The experimental results show that FT-GAN outperforms the strong baselines in SVS on the Gezi Opera synthesis task. Extensive experiments further verify that FT-GAN performs well on synthesis tasks of other operas such as Peking Opera. Audio samples, the dataset, and the codes are available at https://zhengmidon.github.io/FTGAN.github.io/.

show abstract

Section: Experiments Experiments Settingsmentioning

confidence: 99%

“…• Visinger2 (Zhang et al 2022c) Visinger2 is modified from the speech synthesis model VITS (Kim et al 2021). It introduces DDSP (Engel et al 2020) in generating audio to improve performance.…”

Section: Experiments Experiments Settingsmentioning

confidence: 99%

FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis

Zheng,

Bai,

Shi

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Significant progress has been made in the field of Text-Music Generation through various methods, including the use of deep cross-modal correlation learning architectures that determine the similarity between temporal structures in audio and lyrics [5]. One research is VISinger [13],a complete end-to-end high-qualitysinging voice synthesis (SVS) system that directly generates audio waveform from lyrics and musical score. It adopts VAE-based posterior encoder augmented with normalizing flow-based prior encoderand adversarial decoder to realize complete end-to-end speech generation.…”

Section: Text-audio Generationmentioning

confidence: 99%

Research on artificial intelligence generated audio

Liu,

Chang,

Shen

et al. 2023

Sixth International Conference on Computer Information Science and Application Technology (CISAT 2023)

View full text Add to dashboard Cite

AI-generated content(AIGC) has generated widespread media attention with its innovative tools such as ChatGPT and DALL-E, indicating the emergence of a new era of AI.[3] The intense media coverage has provided ample opportunities for the general public to gain insight into AIGC's operations. Audio plays an indispensable role as an information carrier in our daily work and life. With the continuous development of audio generation technology, the audio generation market has huge demand and potential, including music composition, speech synthesis, sound effect generation, and more. However, there remains a deficiency in a comprehensive analysis of artificial intelligence-generated audio (AIGA). As a result, a timely and systematic review of AIGA is necessary to keep up with the unstoppable trend of AI-powered audio creation. Most importantly, our work aims to fill this gap and provide a solid foundation for future research.

show abstract

“…They proposed a shallow diffusion mechanism to improve audio quality and accelerate inference. Visinger (Zhang et al, 2022a) and Visinger2 (Zhang et al, 2022b) are fully endto-end models, and the acoustic model is trained together with the vocoder. This type of model can avoid the problem of error accumulation.…”

Section: Case Studymentioning

confidence: 99%

Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling

Bai,

Zhou,

Zheng

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Singing Voice Synthesis (SVS) strives to synthesize pleasant vocals based on music scores and lyrics. The current acoustic models based on Transformer usually process the entire sequence globally and use a simple L1 loss. However, this approach overlooks the significance of local modeling within the sequence and the local optimization of the hard-to-synthesize parts in the predicted mel-spectrogram. Consequently, the synthesized audio exhibits local incongruities (e.g., local pronunciation jitter or noise). To address this problem, we propose two methods to enhance local modeling in the acoustic model. First, we devise a nearest neighbor local attention, where each phoneme token focuses only on the adjacent phoneme tokens located before and after it. Second, we propose a phoneme-level local adaptive weights loss function that enables the model to focus more on the hard-to-synthesize parts of the mel-spectrogram. We verify the universality of our methods on public Chinese pop song and Hokkien Gezi Opera datasets. Extensive experiments demonstrate the effectiveness of our methods, resulting in significant improvements in both objective and subjective evaluations when compared to the strong baselines. Our code and demonstration samples are available at https://github.com/baipeng1/SVSELM.

show abstract

VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis

Cited by 33 publications

References 10 publications

FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis

FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis

Research on artificial intelligence generated audio

Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling

Contact Info

Product

Resources

About