Fast and High-Quality Singing Voice Synthesis System Based on Convolutional Neural Networks

Abstract

The present paper describes singing voice synthesis based on convolutional neural networks (CNNs). Singing voice synthesis systems based on deep neural networks (DNNs) are currently being proposed and are improving the naturalness of synthesized singing voices. As singing voices represent a rich form of expression, a powerful technique to model them accurately is required. In the proposed technique, long-term dependencies of singing voices are modeled by CNNs. An acoustic feature sequence is generated for each segment that consists of long-term frames, and a natural trajectory is obtained without the parameter generation algorithm. Furthermore, a computational complexity reduction technique, which drives the DNNs in different time units depending on type of musical score features, is proposed. Experimental results show that the proposed method can synthesize natural sounding singing voices much faster than the conventional method.

Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System

“…Although the parameter generation algorithm can generate a smooth acoustic feature sequence, the computational cost at the synthesis stage increases. A recent study [11] introduced a different approach that considers dynamic features only during training. In this approach, the objective function that considers the dynamic features can be written as…”

“…A note pitch transition greatly influences the F0 trajectory. Therefore, we add a skip connection between the input note pitch and a hidden layer of the acoustic model to deliver the note pitch inside the acoustic model, motivated by [11]. This helps transmit the note pitch information efficiently and predict the residual component between log F0 and the note pitch.…”

Sequence-to-Sequence Singing Synthesis Using the Feed-Forward Transformer

“…Non-Seq2Seq singing synthesizers include those based on autoregressive architectures [17,21,22], feed-forward CNN [23], and feed-forward GAN-based approaches [24,25].…”

Sequence-To-Sequence Singing Voice Synthesis With Perceptual Entropy Loss

“…Firstly, the DNN model is proposed to predict the spectral information and begin to outperform conventional HMMs significantly [7,8]. Later on, variations of the neural networks, including Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN), also demonstrate their power on acoustic modeling [6,[9][10][11]. Other architectures, such as the generative adversarial network (GAN), are also shown to improve the synthesized singing quality [12][13][14][15][16][17].…”