A Survey on Recent Deep Learning-driven Singing Voice Synthesis Systems

Yin-Ping, Cho,; Yang, Fu-Rong; Chang, Yung–Chuan; Cheng, Ching-Ting; Wang, Xiaohan; Liu, Yiwen

doi:10.1109/aivr52153.2021.00067

Cited by 13 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Naturally, such systems depend critically on the expressivity of the latent representation and the distribution of training examples, as the decoder may not learn to generate a signal with characteristics that are not observable in the training set. This lack of control is particularly problematic for singing voice signals (Cho et al, 2021), as important expressive parameters like F0 are highly variable within individual notes (Dai and Dixon, 2019) and may not be accurately reproduced by SE or SVS systems, even when F0 is explicitly given in the latent representation (Choi et al, 2022).…”

Section: Singing Voice Reconstructionmentioning

confidence: 99%

A Dataset of Larynx Microphone Recordings for Singing Voice Reconstruction

Schwär,

Krause,

Fast

et al. 2024

Transactions of the International Society for Music Information Retrieval

View full text Add to dashboard Cite

Larynx microphones (LMs) make it possible to obtain practically crosstalk-free recordings of the human voice by picking up vibrations directly from the throat. This can be useful in a multitude of music information retrieval scenarios related to singing, e.g., the analysis of individual voices recorded in environments with lots of interfering noise. However, LMs have a limited frequency range and barely capture the effects of the vocal tract, which makes the recorded signal unsuitable for downstream tasks that require high-quality recordings. In this paper, we introduce the task of reconstructing a natural sounding, high-quality singing voice recording from an LM signal. With an explicit focus on the singing voice, the problem lies at the intersection of speech enhancement and singing voice synthesis with the additional requirement of faithful reproduction of expressive parameters like intonation. In this context, we make three main contributions. First, we publish a dataset with over 4 hours of popular music we recorded with four amateur singers accompanied by a guitar, where both LM and clean close-up microphone signals are available. Second, we propose a data-driven baseline approach for singing voice reconstruction from LM signals using differentiable signal processing, inspired by a source-filter model that emulates the missing vocal tract effects. Third, we evaluate the baseline with a listening test and further show that it can improve the accuracy of lyrics transcription as an exemplary downstream task.

show abstract

Section: Singing Voice Reconstructionmentioning

confidence: 99%

A Dataset of Larynx Microphone Recordings for Singing Voice Reconstruction

Schwär,

Krause,

Fast

et al. 2024

Transactions of the International Society for Music Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Nercessian (2021) incorporated a differentiable harmonic-plusnoise synthesiser (Engel et al, 2020a) to a end-to-end VC model, augmenting it with convolutional pre-and post-nets to further shape the generated signal. This formulation allowed end-to-end training with perceptually informed loss functions, as opposed to requiring autoregression.…”

Section: Voice Transformationmentioning

confidence: 99%

A review of differentiable digital signal processing for music and speech synthesis

Hayes,

Shier,

Fazekas

et al. 2024

Front. Signal Process.

View full text Add to dashboard Cite

The term “differentiable digital signal processing” describes a family of techniques in which loss function gradients are backpropagated through digital signal processors, facilitating their integration into neural networks. This article surveys the literature on differentiable audio signal processing, focusing on its use in music and speech synthesis. We catalogue applications to tasks including music performance rendering, sound matching, and voice transformation, discussing the motivations for and implications of the use of this methodology. This is accompanied by an overview of digital signal processing operations that have been implemented differentiably, which is further supported by a web book containing practical advice on differentiable synthesiser programming (https://intro2ddsp.github.io/). Finally, we highlight open challenges, including optimisation pathologies, robustness to real-world conditions, and design trade-offs, and discuss directions for future research.

show abstract

“…Harshvardhan et al [76] instead covered deep generation as part of generation in machine learning and proposed future directions. In addition to surveys on general deep data generation, other surveys may focus on the deep data generation in specific domains including graph generation [77][78][79], image synthesis [80,81], text generation [82,83] and audio generation [84][85][86].…”

Section: Relationship With Existing Surveysmentioning

confidence: 99%

Controllable Data Generation by Deep Learning: A Review

Wang¹,

Du²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Designing and generating new data under targeted properties has been attracting various critical applications such as molecule design, image editing and speech synthesis. Traditional hand-crafted approaches heavily rely on expertise experience and intensive human efforts, yet still suffer from the insufficiency of scientific knowledge and low throughput to support effective and efficient data generation. Recently, the advancement of deep learning induces expressive methods that can learn the underlying representation and properties of data. Such capability provides new opportunities in figuring out the mutual relationship between the structural patterns and functional properties of the data and leveraging such relationship to generate structural data given the desired properties. This article provides a systematic review of this promising research area, commonly known as controllable deep data generation. Firstly, the potential challenges are raised and preliminaries are provided. Then the controllable deep data generation is formally defined, a taxonomy on various techniques is proposed and the evaluation metrics in this specific domain are summarized. After that, exciting applications of controllable deep data generation are introduced and existing works are experimentally analyzed and compared. Finally, the promising future directions of controllable deep data generation are highlighted and five potential challenges are identified.

show abstract

A Survey on Recent Deep Learning-driven Singing Voice Synthesis Systems

Cited by 13 publications

References 16 publications

A Dataset of Larynx Microphone Recordings for Singing Voice Reconstruction

A Dataset of Larynx Microphone Recordings for Singing Voice Reconstruction

A review of differentiable digital signal processing for music and speech synthesis

Controllable Data Generation by Deep Learning: A Review

Contact Info

Product

Resources

About