Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang, Zhenyu; Bryan, Nicholas J.; Li, Dingzeyu; Langlois, Timothy R.; Manocha, Dinesh

doi:10.1109/tvcg.2020.2973058

Cited by 32 publications

(22 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For current acoustic-based material identification method, Tang et al [20] presented a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. However, they required pre-constructed virtual room geometry as input for acoustic analysis and material optimization, which does not fit our VR/AR application and capability.…”

Section: Related Workmentioning

confidence: 99%

Material Identification System with Sound Simulation Assisted Method in VR/AR Scenarios

Wang

Runting

et al. 2021

Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of The

View full text Add to dashboard Cite

We propose a new model using transfer learning method in motivation of specifying and identifying material of an object by knocking in VR/AR applications. Different from the traditional contact knocking material/object identification method, we apply the sound simulation method to enlarge the training dataset containing various models and materials in real-world scenarios. Our approach is based on Domain-Adversarial Training of Neural Networks that learns from the pre-collected simulated and corresponding real knocking sound to extract their common features determined by different materials. Given the scanned 3D model and the real knocking sound from users, we present an incremental learning model using the features extracted by pre-trained transfer learning model to generate the final material classifier. We perform an overall evaluation showing that our system achieves around 93.3% accuracy of identifying materials, which is much higher than the accuracy mentioned in previous work. CCS CONCEPTS• Human-centered computing → Interaction design.

show abstract

Section: Related Workmentioning

confidence: 99%

Material Identification System with Sound Simulation Assisted Method in VR/AR Scenarios

Wang

Runting

et al. 2021

Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of The

View full text Add to dashboard Cite

show abstract

“…Li et al [23] also generate spatial audio using a recorded acoustic impulse response. Tang et al [41] present a deep learning method to estimate the acoustic material characteristics of rooms and use them for real-world audio rendering. All these methods have been used for real-world rooms and are not directly applicable to arbitrary video recordings.…”

Section: Reconstructing Spatial Information From Audiomentioning

confidence: 99%

Binaural Audio Generation via Multi-task Learning

Li¹,

Liu

Manocha³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Fig. 1. A video with synchronized binaural audio generated by our method. The top row highlights the video frames, and the bottom row highlights the mixed mono audio. The middle two rows are the corresponding left and right channels of the binaural audio. The two audio channels simulate the sounds for the left and right ears, thereby improving the sense of immersion.We present a learning-based approach for generating binaural audio from mono audio using multi-task learning. Our formulation leverages additional information from two related tasks: the binaural audio generation task and the flipped audio classification task. Our learning model extracts spatialization features from the visual and audio input, predicts the left and right audio channels, and judges whether the left and right channels are flipped. First, we extract visual features using ResNet from the video frames. Next, we perform binaural audio generation and flipped audio classification using separate subnetworks based on visual features. Our learning method optimizes the overall loss based on the weighted sum of the losses of the two tasks. We train and evaluate our model on the FAIR-Play dataset and the YouTube-ASMR dataset. We perform quantitative and qualitative evaluations to demonstrate the benefits of our approach over prior techniques.

show abstract

“…The conventional estimation methods use the room impulse response (RIR) [7][8][9] or room transfer function (RTF) [10,11], and convolve RIR and clean sound to create a virtual reverberant sound in a specific space. Among recent studies, there is a study of learning an artificial neural network using the structure of a room and acoustic signals acquired from the room, where RT60, which is an attenuation parameter of the sound pressure level, was estimated and used to construct a reverberant signal [12]. In a similar way, the methods for generating acoustic parameters using deep neural networks (DNNs) have been studied.…”

Section: Introductionmentioning

confidence: 99%

Deep Learning-Based Estimation of Reverberant Environment for Audio Data Augmentation

Yun

Choi

2022

Sensors

View full text Add to dashboard Cite

This paper proposes an audio data augmentation method based on deep learning in order to improve the performance of dereverberation. Conventionally, audio data are augmented using a room impulse response, which is artificially generated by some methods, such as the image method. The proposed method estimates a reverberation environment model based on a deep neural network that is trained by using clean and recorded audio data as inputs and outputs, respectively. Then, a large amount of a real augmented database is constructed by using the trained reverberation model, and the dereverberation model is trained with the augmented database. The performance of the augmentation model was verified by a log spectral distance and mean square error between the real augmented data and the recorded data. In addition, according to dereverberation experiments, the proposed method showed improved performance compared with the conventional method.

show abstract

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Cited by 32 publications

References 58 publications

Material Identification System with Sound Simulation Assisted Method in VR/AR Scenarios

Material Identification System with Sound Simulation Assisted Method in VR/AR Scenarios

Binaural Audio Generation via Multi-task Learning

Deep Learning-Based Estimation of Reverberant Environment for Audio Data Augmentation

Contact Info

Product

Resources

About