Scene-aware audio for 360° videos

Li, Dingzeyu; Langlois, Timothy R.; Zheng, Changxi

doi:10.1145/3197517.3201391

Cited by 47 publications

(58 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analytical gradients can significantly accelerate the optimization process. With similar optimization objectives, it was shown that additional gradient information can boost the speed by a factor of over ten times [33,50]. The speed gain shown by Li et al [33] is impressive, and we further improve the accuracy and speed of the formulation.…”

Section: Related Workmentioning

confidence: 61%

“…With similar optimization objectives, it was shown that additional gradient information can boost the speed by a factor of over ten times [33,50]. The speed gain shown by Li et al [33] is impressive, and we further improve the accuracy and speed of the formulation. More specifically, the original objective function evaluated energy decay relative to the first ray received (the direct sound if there were no obstacles).…”

Section: Related Workmentioning

confidence: 61%

“…With robust room acoustic estimation as input, we present a novel inverse material optimization algorithm to estimate the acoustic properties. We propose a new objective function for material optimization and show that it models the IR decay behavior better than the technique by Li et al [33]. We demonstrate our ability to add new sound sources in regular videos.…”

Section: Introductionmentioning

confidence: 83%

“…Visual cues dominate acoustic cues, so the perceptual difference between how audio sounds and the environment in which it is seen is smaller than the perceived environment of two sounds. Recently, Li et al introduced scene-aware audio to optimize simulator parameters to match the room acoustics from existing recordings [33]. By leveraging visual information for acoustic material classification, Schissler et al demonstrated realistic audio for 3D-reconstructed real-world scenes [50].…”

Section: Related Workmentioning

confidence: 99%

“…In terms of audio "relighting" or reproduction, there have been several approaches proposed toward realistic audio in 360°images [27], multi-modal estimation and optimization [50], and scene-aware audio in 360°videos [33]. However, these approaches either require separate recording of an IR, or produce audio results that are perceptually different from recorded scene audio.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

Fig. 1: Given a natural sound in a real-world room that is recorded using a cellphone microphone (left), we estimate the acoustic material properties and the frequency equalization of the room using a novel deep learning approach (middle). We use the estimated acoustic material properties for generating plausible sound effects in the virtual model of the room (right). Our approach is general and robust, and works well with commodity devices.Abstract-We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real-world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.

show abstract

Section: Related Workmentioning

confidence: 61%

Section: Related Workmentioning

confidence: 61%

Section: Introductionmentioning

confidence: 83%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

show abstract

Inexpensive indoor acoustic material estimation for realistic sound propagation

Kim

Yoon

2022

Computer Animation & Virtual

View full text Add to dashboard Cite

For the realistic and immersive experience in a virtual environment, it is important to estimate and reflect the acoustic characteristic of the real indoor scenes. This article proposes a method of directly measuring the reflection coefficient of a surface, which is an acoustic characteristics in the real environment. Because expensive optimization‐based studies that mainly aim to reproduce recorded sounds indirectly estimate acoustic materials, new estimates are required whenever the actual environment changes. Our approach utilizes the method of the acoustics field to enable anyone to easily and directly measure the reflection coefficient of a real environment and generate sound in a virtual environment. We obtain the impulse response (IR) for the target surface, separate the direct sound and the reflected sound, and calculate the reflection coefficient for each surface. The measurement of the IR and the reflection coefficient calculation takes about 2 s. Our result produces a sound with a similar reverberation time to the sound recorded in the real environment.

show abstract

Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds

Vasudevan

Dai

Gool

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Humans can robustly recognize and localize objects by integrating visual and auditory cues. While machines are able to do the same now with images, less work has been done with sounds. This work develops an approach for dense semantic labelling of sound-making objects, purely based on binaural sounds. We propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360 • camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framework that consists of a vision 'teacher' method and a sound 'student' method -the student method is trained to generate the same results as the teacher method. This way, the auditory system can be trained without using human annotations. We also propose two auxiliary tasks namely, a) a novel task on Spatial Sound Super-resolution to increase the spatial resolution of sounds, and b) dense depth prediction of the scene. We then formulate the three tasks into one end-to-end trainable multi-tasking network aiming to boost the overall performance. Experimental results on the dataset show that 1) our method achieves good results for all the three tasks; and 2) the three tasks are mutually beneficial -training them together achieves the best performance and 3) the number and the orientations of microphones are both important. The data and code will be released on the project page.

show abstract

Scene-aware audio for 360° videos

Cited by 47 publications

References 30 publications

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Inexpensive indoor acoustic material estimation for realistic sound propagation

Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds

Contact Info

Product

Resources

About