Reverb Conversion of Mixed Vocal Tracks Using an End-to-end Convolutional Deep Neural Network

Abstract

Reverb plays a critical role in music production, where it provides listeners with spatial realization, timbre, and texture of the music. Yet, it is challenging to reproduce the musical reverb of a reference music track even by skilled engineers. In response, we propose an end-to-end system capable of switching the musical reverb factor of two different mixed vocal tracks. This method enables us to apply the reverb of the reference track to the source track to which the effect is desired. Further, our model can perform de-reverberation when the reference track is used as a dry vocal source. The proposed model is trained in combination with an adversarial objective, which makes it possible to handle high-resolution audio samples. The perceptual evaluation confirmed that the proposed model can convert the reverb factor with the preferred rate of 64.8%. To the best of our knowledge, this is the first attempt to apply deep neural networks to converting music reverb of vocal tracks.

Filtered Noise Shaping for Time Domain Room Impulse Response Estimation From Reverberant Speech

“…For the task of acoustic matching for audio post-production, Sarroff and Michaels [11] proposed an RNN trained to predict the parameters of an algorithmic reverb given a reverberant recording. Koo et al [12] extended this with a U-Net trained to directly convert singing voice recordings to match a target signal, forgoing the need to estimate the parameters of an algorithmic reverb. Nevertheless, while these results are convincing, both methods rely upon training data generated with algorithmic reverbs, which may limit generalization for matching real environments in augmented reality.…”

“…While these parameters provide useful information about room acoustics, they can be limiting in cases where auralization of the RIR is required. This is especially the case with augmented and virtual reality (AR and VR), and is also the case with reverb matching procedures for audio post-production [9][10][11][12]. In these cases, accurately matching the characteristics of the room acoustics is generally required, which often extends beyond T60 and DRR.…”

“…Several deep learning approaches have been introduced to address this acoustic matching task. Many are focused on audio post-production, and therefore rely on training data generated with algorithmic reverbs that may not generalize to real rooms [9,11,12]. Other approaches are trained using measured RIRs and additionally handle removing room effects from the source recordings, but operate at lower sample rates, and employ a large model for the transformation, which can be compute intensive during inference [10].…”