A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

Abstract

In this publication, we present a deep learning-based method to transform the f0 in speech and singing voice recordings. f0 transformation is performed by training an auto-encoder on the voice signal’s mel-spectrogram and conditioning the auto-encoder on the f0. Inspired by AutoVC/F0, we apply an information bottleneck to it to disentangle the f0 from its latent code. The resulting model successfully applies the desired f0 to the input mel-spectrograms and adapts the speaker identity when necessary, e.g., if the requested f0 falls out of the range of the source speaker/singer. Using the mean f0 error in the transformed mel-spectrograms, we define a disentanglement measure and perform a study over the required bottleneck size. The study reveals that to remove the f0 from the auto-encoder’s latent code, the bottleneck size should be smaller than four for singing and smaller than nine for speech. Through a perceptive test, we compare the audio quality of the proposed auto-encoder to f0 transformations obtained with a classical vocoder. The perceptive test confirms that the audio quality is better for the auto-encoder than for the classical vocoder. Finally, a visual analysis of the latent code for the two-dimensional case is carried out. We observe that the auto-encoder encodes phonemes as repeated discontinuous temporal gestures within the latent code.

Analysis and transformations of intensity in singing voice

“…We can use a bottleneck auto-encoder [14] to disentangle the voice intensity from the mel-spectrogram of singing voice recordings. We extend the architecture of [16] to additionally include the voice intensity as conditional input.…”

“…A detailed description of this bottleneck auto-encoder is given in [16] so we only give a brief outline here: The autoencoder consists of a pair of networks, an encoder and a decoder which are cascaded (the input of the decoder is the output of the encoder). Additionally the decoder receives a conditional input, which in this case consists of the f0, voiced-unvoiced mask and the intensity from Section 2.…”

“…All experiments are trained on the same dataset as [16], which is a combined dataset of CREL Research Database (SVDB) [18], NUS sung and spoken lyrics corpus [19], from the i-Treasures Intangible Cultural Heritage dataset [20] PJS phoneme-balanced Japanese singing-voice corpus [21], JVS-MuSiC [22], Tohoku Kiritan and Itako singing database [23], VocalSet: A singing voice dataset [24], as well as singing recordings from our internal singing databases used for the IR-CAM Singing Synthesizer [25] and other projects.…”

“…We train a neural network to disentangle the intensity from the signal's mel-spectrogram. Neural disentanglement has received large attention in recent years and has been applied for example to speaker identity [14,15] and fundamental frequency [16].…”

“…The remaining paper is structured as follows: In Section 2 we introduce two ways to estimate the relative voice intensity from voice recordings where no intensity annotation exists. In Section 3 we present our adaptions of [16] to include the voice intensity as a controllable parameter. We will explain our experimental validation in Section 4 and present and discuss the results in Section 5.…”