BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis

Abstract

Binaural audio plays a significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasing attention. This synthesis process involves not only the basic physical warping of the mono audio, but also room reverberations and head/ear related filtrations, which, however, are difficult to accurately simulate in traditional digital signal processing. In this paper, we formulate the synthesis process from a different perspective by decomposing the binaural audio into a common part that shared by the left and right channels as well as a specific part that differs in each channel. Accordingly, we propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Specifically, in the first stage, the common information of the binaural audio is generated with a single-channel diffusion model conditioned on the mono audio, based on which the binaural audio is generated by a two-channel diffusion model in the second stage. Combining this novel perspective of two-stage synthesis with advanced generative models (i.e., the diffusion models), the proposed BinauralGrad is able to generate accurate and high-fidelity binaural audio samples. Experiment results show that on a benchmark dataset, BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics (Wave L2: 0.128 vs. 0.157, MOS: 3.80 vs. 3.61). The generated audio samples are available online 3 .

DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech — A Study Between English and Mandarin

“…One major drawback of DPM-based models is the slow sampling speed due to many iterative steps. Therefore, many previous DPM-based TTS methods focus on accelerating the sampling method to boost the inference speed [38], [39], [56], [57], [58]. Some research considers changing the training process to generate high-quality speech.…”

ERA-Solver: Error-Robust Adams Solver for Fast Sampling of Diffusion Probabilistic Models

“…Denoising diffusion probabilistic models (DDPMs) (Ho et al, 2020) have demonstrated their great generation potential on various applications, such as text-to-image synthesis (Poole et al, 2022;Gu et al, 2022;Kim & Ye, 2021), image inpainting (Lugmayr et al, 2022;Liu et al, 2022;Kawar et al, 2022), speech synthesis (Huang et al, 2021;Lam et al, 2022;Leng et al, 2022), and molecular conformation generation (Hoogeboom et al, 2022;Jing et al, 2022;Wu et al, 2022;Huang et al, 2022). It involves a diffusion process to gradually add noise to data, and a parameterized denoising process to reverse the diffusion process, sampling through gradually removing the noise from random noise.…”

“…In recent years, denoising diffusion probabilistic models (DDPMs) (Ho et al, 2020) have been proven to have potential in data generation tasks such as text-to-image generation (Poole et al, 2022;Gu et al, 2022;Kim & Ye, 2021;Chen et al, 2022), speech synthesis (Huang et al, 2021;Lam et al, 2022;Leng et al, 2022), and molecular conformation formation (Hoogeboom et al, 2022;Jing et al, 2022;Wu et al, 2022;Huang et al, 2022). They build a diffusion process to add noise into the sample and a denoising process to remove noise from the sample gradually.…”

An Context-Aware Intelligent System to Automate the Conversion of 2D Audio to 3D Audio using Signal Processing and Machine Learning

“…In our approach, we decided that rather than relying on visual cues, allowing direct customization of the output 3D signal with an interactive interface would allow for more freedom and generate more satisfactory results. This paper presented a synthesis process in order to generate binaural audio from a mono audio source, also similar to what we are trying to accomplish [10]. Instead of utilizing head-related transfer functions, they created a novel process for generating binaural audio utilizing diffusion models.…”