Flow Synthesizer: Universal Audio Synthesizer Control with Normalizing Flows

Abstract

The ubiquity of sound synthesizers has reshaped music production and even entirely defined new music genres. However, the increasing complexity and number of parameters in modern synthesizers make them harder to master. Hence, the development of methods allowing to easily create and explore with synthesizers is a crucial need.Here, we introduce a novel formulation of audio synthesizer control. We formalize it as finding an organized latent audio space that represents the capabilities of a synthesizer, while constructing an invertible mapping to the space of its parameters. By using this formulation, we show that we can address simultaneously automatic parameter inference, macro-control learning and audio-based preset exploration within a single model. To solve this new formulation, we rely on Variational Auto-Encoders (VAE) and Normalizing Flows (NF) to organize and map the respective auditory and parameter spaces. We introduce the disentangling flows, which allow to perform the invertible mapping between separate latent spaces, while steering the organization of some latent dimensions to match target variation factors by splitting the objective as partial density evaluation. We evaluate our proposal against a large set of baseline models and show its superiority in both parameter inference and audio reconstruction. We also show that the model disentangles the major factors of audio variations as latent dimensions, that can be directly used as macro-parameters. We also show that our model is able to learn semantic controls of a synthesizer by smoothly mapping to its parameters. Finally, we discuss the use of our model in creative applications and its realtime implementation in Ableton Live 1 .

Transfer Learning of the Expressivity Using FLOW Metric Learning in Multispeaker Text-to-Speech Synthesis

“…We used 4 speech corpora for developing our multispeaker expressive TTS system. The speech corpora are Lisa [12], a French female neutral corpus (approx. 3 hrs), Caroline [20], a French female expressive corpus (approx.…”

“…For fast and high-fidelity wavenet based speech synthesis, the authors in [11] proposed probability density distillation to fill in the bridge between trained Wavenet as teacher model and IAF as a student model. In [12], the authors proposed a universal audio synthesizer built using normalizing flows to learn the latent space representation for semantic control of a synthesizer by interpolation of latent variables. In this paper, IAF were used as normalizing flows to perform the variational inference for learning meaningful latent space representation .…”

On-The-Fly Syntheziser Programming with Fuzzy Rule Learning

“…For the SVM, tested parameter values for C and gamma were respectively [0.01, 0.1, 1, 10, 100, 1000] and [1, 0.1, 0.01, 0.001, 0.00001, 0.000001, 10]. For KNN, the tested N values were [1,2,3,4,5,6,7,8,9,10] and for the Random forest [1, 10, 100, 500,1000] trees were considered. In the case of the FuzzyRuLer, d was explored from 1 to half the number of features in the dataset and ratio with [0.9, 0.8, 0.7, 0.6, 0.5] values.…”

“…In [ 8 ], a methodology is presented that relates the spaces of parameters and audio capabilities of a synthesizer in such a way that the mapping relating those spaces is invertible, which encourages high-level interactions with the synth. The system allows intuitive audio-based preset exploration.…”

Special Issue on Digital Audio Effects

“…Similarly to effects processing, sound synthesis has been a core subject at DAFx conferences from the beginning. Flow Synthesizer: Universal Audio Synthesizer Control with Normalizing Flows by Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, and Axel Chemla-Romeu-Santos [12] highlights the importance of affordable, intuitive control in today's complex sound synthesizers. By using Variational Auto-Encoders and Normalizing Flows, the authors invert the synthesizer's map from its controls to a "latent" audio space, this way inferring parameters, learning control, drawing smooth transitions, and defining presets.…”