MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes

Abstract

We propose a mesh-based neural network (MESH2IR) to generate acoustic impulse responses (IRs) for indoor 3D scenes represented using a mesh. The IRs are used to create a high-quality sound experience in interactive applications and audio processing. Our method can handle input triangular meshes with arbitrary topologies (2K -3M triangles). We present a novel training technique to train MESH2IR using energy decay relief and highlight its benefits. We also show that training MESH2IR on IRs preprocessed using our proposed technique significantly improves the accuracy of IR generation. We reduce the non-linearity in the mesh space by transforming 3D scene meshes to latent space using a graph convolution network. Our MESH2IR is more than 200 times faster than a geometric acoustic algorithm on a CPU and can generate more than 10,000 IRs per second on an NVIDIA GeForce RTX 2080 Ti GPU for a given furnished indoor 3D scene. The acoustic metrics are used to characterize the acoustic environment. We show that the acoustic metrics of the IRs predicted from our MESH2IR match the ground truth with less than 10% error. We also highlight the benefits of MESH2IR on audio and speech processing applications such as speech dereverberation and speech separation. To the best of our knowledge, ours is the first neural-network-based approach to predict IRs from a given 3D scene mesh in real-time.

Real-time Auralization based on a Convolutional Neural Network Trained by Low-Frequency Wave-Based Calculations

“…[8] Notably, Borrel-Jensen et al presented a physics-informed neural network to predict the solution to the linear wave equation to obtain the sound field in 1D with parameterized sources and impedance boundaries, and this method will be further applied in realistic 3D scenes. [9] In addition, there is a mesh-based neural network to generate impulse responses (IRs) for indoor 3D scenes whereby 3D scene meshes were transformed into latent space and the latent space was used to generate IRs, [10] and a Neural Acoustic Fields methodology that represents how sounds propagate in a physical scene and learns to continuously map all emitter and listener location pairs to a neural impulse response function. [11] Besides, NNs are used to predict reverberation time, room volume, absorption coefficients, and eigenfrequencies, and analyze multiexponential sound energy decay.…”