Automatic Programming of VST Sound Synthesizers Using Deep Networks and Other Techniques

Yee-King, Matthew; Fedden, Leon; d’Inverno, Mark

doi:10.1109/tetci.2017.2783885

Cited by 31 publications

(54 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the past years, the automatic parameterization of synthesizers has been the subject of several studies [3,4,16]. All of these approaches share the objective to optimize the correspondence between the generated sound and a given target sound.…”

Section: Synthesizer Parameters Optimizationmentioning

confidence: 99%

“…An alternative to manual parameters setting would be to infer the set of parameters that could best reproduce a given target sound. This task of parameters inference has been studied in the past years using various techniques, such as iterative relevance feedback on audio descriptors [2], Genetic Programming to directly grow modular synthesizers [3], or bi-directional LSTM with highway layers [4] to produce parameters approximation. Although these approaches might be appealing, they all share the same fundamental flaws that (i) though it is unlikely that a synthesizer can generate exactly any audio target, none explicitly model these limitations, (ii) they do not account for the non-linear relationships that exist between parameters and the corresponding synthesized audio, and (iii) none of these approaches allow for higher-level controls or interaction with audio synthesizers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Flow Synthesizer: Universal Audio Synthesizer Control with Normalizing Flows

et al. 2019

View full text Add to dashboard Cite

The ubiquity of sound synthesizers has reshaped music production and even entirely defined new music genres. However, the increasing complexity and number of parameters in modern synthesizers make them harder to master. Hence, the development of methods allowing to easily create and explore with synthesizers is a crucial need.Here, we introduce a novel formulation of audio synthesizer control. We formalize it as finding an organized latent audio space that represents the capabilities of a synthesizer, while constructing an invertible mapping to the space of its parameters. By using this formulation, we show that we can address simultaneously automatic parameter inference, macro-control learning and audio-based preset exploration within a single model. To solve this new formulation, we rely on Variational Auto-Encoders (VAE) and Normalizing Flows (NF) to organize and map the respective auditory and parameter spaces. We introduce the disentangling flows, which allow to perform the invertible mapping between separate latent spaces, while steering the organization of some latent dimensions to match target variation factors by splitting the objective as partial density evaluation. We evaluate our proposal against a large set of baseline models and show its superiority in both parameter inference and audio reconstruction. We also show that the model disentangles the major factors of audio variations as latent dimensions, that can be directly used as macro-parameters. We also show that our model is able to learn semantic controls of a synthesizer by smoothly mapping to its parameters. Finally, we discuss the use of our model in creative applications and its realtime implementation in Ableton Live 1 .

show abstract

Section: Synthesizer Parameters Optimizationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Flow Synthesizer: Universal Audio Synthesizer Control with Normalizing Flows

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Nevertheless, this approach did not yield any improvement over our final models that are presented in this paper. Finally, in contrast to both [27,28] as well as [10], we do not solve a regression task and formulate the parameter estimation task as The STFT spectrogram of the input signal is fed into a 2D CNN that predicts the synthesizer parameter configuration. This configuration is then used to produce a sound that is similar to the input sound.…”

Section: Related Workmentioning

confidence: 99%

InverSynth: Deep Estimation of Synthesizer Parameter Configurations From Audio Signals

Barkan

Tsiris

Katz

et al. 2019

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Sound synthesis is a complex field that requires domain expertise. Manual tuning of synthesizer parameters to match a specific sound can be an exhaustive task, even for experienced sound engineers. In this paper, we introduce InverSynth -an automatic method for synthesizer parameters tuning to match a given input sound. InverSynth is based on strided convolutional neural networks and is capable of inferring the synthesizer parameters configuration from the input spectrogram and even from the raw audio. The effectiveness InverSynth is demonstrated on a subtractive synthesizer with four frequency modulated oscillators, envelope generator and a gater effect. We present extensive quantitative and qualitative results that showcase the superiority InverSynth over several baselines. Furthermore, we show that the network depth is an important factor that contributes to the prediction accuracy.Index Terms -deep synthesizer parameter estimation, automatic sound synthesis, inverse problems, InverSynthParts of this manuscript are based on a previous publication [26].

show abstract

“…They are widely used for music information retrieval tasks (Hamel and Eck, 2010) across a variety of use cases (e.g. (Yee-King et al, 2018;Khunarsal et al, 2013).…”

Section: Error and Similarity Metricsmentioning

confidence: 99%

Peer Review #3 of "Sample-level sound synthesis with recurrent neural networks and conceptors (v0.1)"

Stoller¹

2019

View full text Add to dashboard Cite

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with generation of long sequences (of more than 20 samples) and multi-timbre pattern generation. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined. INTRODUCTION 33 Machine Learning and Sound Synthesis 34 Current intersections between sound synthesis and machine learning are evolving quickly. We have seen 35 significant progress in symbolic note generation (e.g. RL Tuner (Jaques et al., 2016), Flow Machines 36 (Ghedini et al., 2016)), parametric control of sound synthesis models (e.g Wekinator (Fiebrink, 2011), 37 automatic VST programming (Yee-King et al., 2018)) and also with current state of the art raw audio 38 generation techniques. These recent advances in raw audio synthesis principally use deep architectures, 39

show abstract

Automatic Programming of VST Sound Synthesizers Using Deep Networks and Other Techniques

Cited by 31 publications

References 21 publications

Flow Synthesizer: Universal Audio Synthesizer Control with Normalizing Flows

Flow Synthesizer: Universal Audio Synthesizer Control with Normalizing Flows

InverSynth: Deep Estimation of Synthesizer Parameter Configurations From Audio Signals

Peer Review #3 of "Sample-level sound synthesis with recurrent neural networks and conceptors (v0.1)"

Contact Info

Product

Resources

About