Speech Synthesis and Control Using Differentiable DSP

Abstract

Modern text-to-speech systems are able to produce natural and highquality speech, but speech contains factors of variation (e.g. pitch, rhythm, loudness, timbre) that text alone cannot contain. In this work we move towards a speech synthesis system that can produce diverse speech renditions of a text by allowing (but not requiring) explicit control over the various factors of variation. We propose a new neural vocoder that offers control of such factors of variation. This is achieved by employing differentiable digital signal processing (DDSP) (previously used only for music rather than speech), which exposes these factors of variation. The results show that the proposed approach can produce natural speech with realistic timbre, and individual factors of variation can be freely controlled.

Controllable Data Generation by Deep Learning: A Review

“…In addition to the proposed taxonomy, we summarize the peer-reviewed publications following the aforementioned taxonomy in Table 1. Post-generation control Optimiation in latent space Optimization in data space [31,130,[169][170][171][172] Peri-generation control Goal-oriented control [1,3,7,[173][174][175][176][177][178][179][180][181]] Distribution-guided control [10,182,183] Controllable transformation from source data Fixed transformation Within-domain transformation [171,[184][185][186][187][188][189][190][191]] Cross-domain transformation [185,186,188,192,193] Steerable transformation Control via reference data [37,] Control via latent space [11,[215][216][217][218][219][220][221][222]…”

“…EMOVIE achieves TTS transformation while controlling the emotion via the emotion embedding supervised by the emotion labels [229]. Fabbro et al [215] controls the TTS systems by decomposing the spectrogram into control variables, such as amplitude envelope, harmonic distribution, and filter coefficients. SCTKG generates essays from given topics while controlling sentiment for each sentence by injecting the sentiment information into the generator [224].…”

“…Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours [304]. LJSpeech dataset is commonly used in controllable TTS-related tasks [192,226] or simple speech synthesis [215]. LibriTTS dataset is a multi-speaker English corpus of approximately 585 hours of read English speech at 24kHz sampling rate, and was specified employed in controllable TTS tasks [203,212].…”

Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfer

“…More recently, a complementary class of vocoder algorithms including source-filter models [20] and differentiable digital signal processing (DDSP) [3] explicitly model monophonic source production in a parametric manner, hearkening back to the core signal processing approaches of the past. Such techniques have been successfully extended to speech applications, either as standalone vocoders [10,4] or as part of an end-to-end system [14]. However, the main disadvantage of these approaches is that they lack a deterministic analysis procedure to accurately extract vocoder parameters directly from source audio.…”