Automatic Multitrack Mixing With A Differentiable Mixing Console Of Neural Audio Effects

Abstract

Applications of deep learning to automatic multitrack mixing are largely unexplored. This is partly due to the limited available data, coupled with the fact that such data is relatively unstructured and variable. To address these challenges, we propose a domain-inspired model with a strong inductive bias for the mixing task. We achieve this with the application of pre-trained sub-networks and weight sharing, as well as with a sum/difference stereo loss function. The proposed model can be trained with a limited number of examples, is permutation invariant with respect to the input ordering, and places no limit on the number of input sources. Furthermore, it produces human-readable mixing parameters, allowing users to manually adjust or refine the generated mix. Results from a perceptual evaluation involving audio engineers indicate that our approach generates mixes that outperform baseline approaches. To the best of our knowledge, this work demonstrates the first approach in learning multitrack mixing conventions from real-world data at the waveform level, without knowledge of the underlying mixing parameters.

End-to-end Music Remastering System Using Self-supervised and Adversarial Training

“…Following this convention, we designed the Mastering Effects Manipulator as the most basic mastering chain by implementing parametric equalizer (EQ) for tonal control, multiband stereo imager for spatial adjustment, and maximizer (or compressor) for voluminous control. For the gain controller, EQ, and maximizer, we follow the implementation to pymixconsole 1 -the collection of audio effects modules presented at [10], and implement the multiband stereo imager that manipulates stereo width in each band using Linkwitz-Riley crossover filter [11].…”

“…For LMSS, we utilize the original implementation of [18] with FFT sizes of (4096, 2048, 1024, 512) and apply it to both stereo-channeled and mid/side-channeled outputs. Computing mid/side-channeled outputs with multi-scale spec-tral objective has been proposed as the stereo loss function in [10], which is to separately calculate summation and subtraction of left and right channels with LMSS so that the phase-related information can also be addressed. The total objective function for the Mastering Cloner is as follow:…”

“…This is in comparison to the music representation from [13], where the input signal of mono-channeled and a sampling rate of 16 kHz is encoded to the dimensionality of 512, which may restrain its performance upon audio effects. Furthermore, since the network is capable of encoding variable-length inputs into features of fixed size, we randomly segmentized input audio with a duration of [5,10] seconds. We set the batch size to 380 using multiple GPUs and trained it for 100 epochs with the Adam optimizer [23] of a learning rate of 2 • 10 −4 .…”

Style Transfer of Audio Effects with Differentiable Signal Processing

“…However, this approach requires manual implementation and often modification of DSP, imparting high engineering cost, potentially limiting its application. To work around this, two alternative methods have been proposed including neural proxies (NP) of audio effects [16] and numerical gradient approximation schemes [17]. However, given that these approaches were proposed and evaluated in different tasks, it is difficult to fully understand their relative performance.…”

Removing Distortion Effects in Music Using Deep Neural Networks

“…unprocessed) source tracks are required. Given the recent advances in automatic mixing [4,5] and music source separation [6], a system could be developed that facilitates the adjustment of a stereo mixture to the user's taste and preferences similar to [7]. However, today's source separation systems are commonly trained on data that is based on music stems (e.g., MUSDB18 [8]).…”