MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes

Ratnarajah, Anton; Tang, Zhenyu; Aralikatti, Rohith; Manocha, Dinesh

doi:10.1145/3503161.3548253

Cited by 14 publications

(5 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In ref. 25, a 3D scene mesh is encoded to a latent space and combined with source/receiver pairs to produce an embedding that is passed through a generative adversarial network for IR generation; frameworks predicting binaural IRs inputting source/receiver pairs, head orientation, and time are presented in refs. 26 and 27, where the former is taking into account a local 2D feature grid representing a static scene, whereas the latter is motivated by acoustic radiance transfer considering surface points for static scenes; in ref.…”

Section: Significancementioning

confidence: 99%

Sound propagation in realistic interactive 3D scenes with parameterized sources using deep neural operators

Borrel-Jensen,

Goswami,

Engsig-Karup

et al. 2024

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

We address the challenge of acoustic simulations in three-dimensional (3D) virtual rooms with parametric source positions, which have applications in virtual/augmented reality, game audio, and spatial computing. The wave equation can fully describe wave phenomena such as diffraction and interference. However, conventional numerical discretization methods are computationally expensive when simulating hundreds of source and receiver positions, making simulations with parametric source positions impractical. To overcome this limitation, we propose using deep operator networks to approximate linear wave-equation operators. This enables the rapid prediction of sound propagation in realistic 3D acoustic scenes with parametric source positions, achieving millisecond-scale computations. By learning a compact surrogate model, we avoid the offline calculation and storage of impulse responses for all relevant source/listener pairs. Our experiments, including various complex scene geometries, show good agreement with reference solutions, with root mean squared errors ranging from 0.02 to 0.10 Pa. Notably, our method signifies a paradigm shift as—to our knowledge—no prior machine learning approach has achieved precise predictions of complete wave fields within realistic domains.

show abstract

Section: Significancementioning

confidence: 99%

Sound propagation in realistic interactive 3D scenes with parameterized sources using deep neural operators

Borrel-Jensen,

Goswami,

Engsig-Karup

et al. 2024

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Because the environment, emitter and receiver are static, so are the acoustics. Other work predicts impulse responses in simulation either for a single environment [28], or by using few-shot egocentric observations [30], or by using the 3D scene mesh [44]. While simulated results are satisfying, those models' impact on real-world data is unknown, especially for scenarios where human speakers move and interact with each other.…”

Section: Related Workmentioning

confidence: 99%

Novel-View Acoustic Synthesis

Chen¹,

Richard²,

Шаповалов³

et al. 2023

Preprint

View full text Add to dashboard Cite

We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the sound of that scene from an unseen target viewpoint? We propose a neural rendering approach: Visually-Guided Acoustic Synthesis (ViGAS) network that learns to synthesize the sound of an arbitrary point in space by analyzing the input audio-visual cues.To benchmark this task, we collect two first-of-their-kind large-scale multi-view audio-visual datasets, one synthetic and one real. We show that our model successfully reasons about the spatial cues and synthesizes faithful audio on both datasets. To our knowledge, this work represents the very first formulation, dataset, and approach to solve the novelview acoustic synthesis task, which has exciting potential applications ranging from AR/VR to art and design. Unlocked by this work, we believe that the future of novel-view synthesis is in multi-modal learning from videos.

show abstract

“…In recent years, generative models, particularly generative adversarial networks (GANs) [21], have gained significant attention and success in various domains, including audio super-resolution and RIR synthesis [22][23][24][25]. The GAN framework consists of a generator and a discriminator.…”

Section: Introductionmentioning

confidence: 99%

Upmix B-Format Ambisonic Room Impulse Responses Using a Generative Model

Xia,

Zhang

2023

Applied Sciences

View full text Add to dashboard Cite

Ambisonic room impulse responses (ARIRs) are recorded to capture the spatial acoustic characteristics of specific rooms, with widespread applications in virtual and augmented reality. While the first-order Ambisonics (FOA) microphone array is commonly employed for three-dimensional (3D) room acoustics recording due to its easy accessibility, higher spatial resolution necessitates using higher-order Ambisonics (HOA) in applications such as binaural rendering and sound field reconstruction. This paper introduces a novel approach, leveraging generative models to upmix ARIRs. The evaluation results validate the model’s effectiveness at upmixing first-order ARIRs to higher-order representations, surpassing the aliasing frequency limitations. Furthermore, the spectral errors observed in the Binaural Room Transfer Functions (BRTFs) indicate the potential benefits of using upmixed ARIRs for binaural rendering, significantly improving rendering accuracy.

show abstract

MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes

Cited by 14 publications

References 24 publications

Sound propagation in realistic interactive 3D scenes with parameterized sources using deep neural operators

Sound propagation in realistic interactive 3D scenes with parameterized sources using deep neural operators

Novel-View Acoustic Synthesis

Upmix B-Format Ambisonic Room Impulse Responses Using a Generative Model

Contact Info

Product

Resources

About