Speech-To-Singing Conversion in an Encoder-Decoder Framework

Parekh, Jayneel; Rao, Preeti; Yang, Yi-Hsuan

doi:10.1109/icassp40776.2020.9054473

Cited by 12 publications

(23 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…al [23] proposed voice conversion approach to transform speech to singing style. An encoder-decoder approach to model-based learning of speech to singing voice conversion was explored in Parekh [24]. However many of the past methods are not scalable to large volumes of speech data needed for acoustic model training in a E2E lyrics transcription system.…”

Section: Related Prior Workmentioning

confidence: 99%

End-to-End Lyrics Recognition with Voice to Singing Style Transfer

Basak

Agarwal

Ganapathy

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Automatic transcription of monophonic/polyphonic music is a challenging task due to the lack of availability of large amounts of transcribed data. In this paper, we propose a data augmentation method that converts natural speech to singing voice based on vocoder based speech synthesizer. This approach, called voice to singing (V2S), performs the voice style conversion by modulating the F0 contour of the natural speech with that of a singing voice. The V2S model based style transfer can generate good quality singing voice thereby enabling the conversion of large corpora of natural speech to singing voice that is useful in building an E2E lyrics transcription system. In our experiments on monophonic singing voice data, the V2S style transfer provides a significant gain (relative improvements of 21 %) for the E2E lyrics transcription system. We also discuss additional components like transfer learning and lyrics based language modeling to improve the performance of the lyrics transcription system.

show abstract

Section: Related Prior Workmentioning

confidence: 99%

End-to-End Lyrics Recognition with Voice to Singing Style Transfer

Basak

Agarwal

Ganapathy

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Speech-to-sing (STS) can be regarded as an example of the general problem of style transfer. Compared with other style transfer work, the "style" of STS not only refers to the incorporation of the required melody, but also requires maintenance of the speaker's identity while transferring the timbre of speech to that of singing [274].…”

Section: Speech-to-sing(sts)mentioning

confidence: 99%

“…The algorithm was evaluated on the task of SVC, and results showed that the algorithm can produce high-quality singing voice which is highly similar to the target speaker's voice given the normal speech samples of the target speaker. Parekh et al [274] explored a method to achieve STS conversion by employing the minimal additional information over the melody contour.…”

Section: Speech-to-sing(sts)mentioning

confidence: 99%

“…All singing recordings have been phonetically transcribed with duration boundaries, and the total number of annotated phones is 25,474. NUS-48E corpus is the first phonetically annotated dataset for singing voice research, and the largest paired dataset for Speech-to-Singing (STS) tasks [274]. Plenty of SVC studies have also employed this dataset [267,270].…”

Section: Audiomentioning

confidence: 99%

See 1 more Smart Citation

A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions

Ji,

Luo,

Yang

2020

Preprint

View full text Add to dashboard Cite

The utilization of deep learning techniques in generating various contents (such as image, text, etc.) has become a trend. Especially music, the topic of this paper, has attracted widespread attention of countless researchers.The whole process of producing music can be divided into three stages, corresponding to the three levels of music generation: score generation produces scores, performance generation adds performance characteristics to the scores, and audio generation converts scores with performance characteristics into audio by assigning timbre or generates music in audio format directly. Previous surveys have explored the network models employed in the field of automatic music generation. However, the development history, the model evolution, as well as the pros and cons of same music generation task have not been clearly illustrated. This paper attempts to provide an overview of various composition tasks under different music generation levels, covering most of the currently popular music generation tasks using deep learning. In addition, we summarize the datasets suitable for diverse tasks, discuss the music representations, the evaluation methods as well as the challenges under different levels, and finally point out several future directions.

show abstract

“…Following the convention in the literature, we define unconditional generation as a task that aims at generating things from scratch, i.e., taking nothing but random noises as the input. In contrast, a conditional generation model takes additional input such as class labels[8], text[9], pitch labels[10,11], or reference audio[12,13,14].…”

mentioning

confidence: 99%

Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization

Liu,

Chen,

Yeh

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In a recent paper, we have presented a generative adversarial network (GAN)-based model for unconditional generation of the mel-spectrograms of singing voices. As the generator of the model is designed to take a variable-length sequence of noise vectors as input, it can generate mel-spectrograms of variable length. However, our previous listening test shows that the quality of the generated audio leaves room for improvement. The present paper extends and expands that previous work in the following aspects. First, we employ a hierarchical architecture in the generator to induce some structure in the temporal dimension. Second, we introduce a cycle regularization mechanism to the generator to avoid mode collapse. Third, we evaluate the performance of the new model not only for generating singing voices, but also for generating speech voices. Evaluation result shows that new model outperforms the prior one both objectively and subjectively. We also employ the model to unconditionally generate sequences of piano and violin music and find the result promising. Audio examples, as well as the code for implementing our model, will be publicly available online upon paper publication.

show abstract

Speech-To-Singing Conversion in an Encoder-Decoder Framework

Cited by 12 publications

References 28 publications

End-to-End Lyrics Recognition with Voice to Singing Style Transfer

End-to-End Lyrics Recognition with Voice to Singing Style Transfer

A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions

Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization

Contact Info

Product

Resources

About