Machine-learning-based estimation and rendering of scattering in virtual reality

Abstract

In this work, a technique to render the acoustic effect of scattering from finite objects in virtual reality is proposed, which aims to provide a perceptually plausible response for the listener, rather than a physically accurate response. The effect is implemented using parametric filter structures and the parameters for the filters are estimated using artificial neural networks. The networks may be trained with modeled or measured data. The input data consist of a set of geometric features describing a large quantity of source-object-receiver configurations, and the target data consist of the filter parameters computed using measured or modeled data. A proof-of-concept implementation is presented, where the geometric descriptions and computationally modeled responses of three-dimensional plate objects are used for training. In a dynamic test scenario, with a single source and plate, the approach is shown to provide a similar spectrogram when compared with a reference case, although some spectral differences remain present. Nevertheless, it is shown with a perceptual test that the technique produces only a slightly lower degree of plausibility than the state-of-the-art acoustic scattering model that accounts for diffraction, and also that the proposed technique yields a prominently higher degree of plausibility than a model that omits diffraction.

Sound Synthesis, Propagation, and Rendering: A Survey

“…Pulkki1 and Svensson [157] proposed a machine learning method for estimating filter-parameters from the object geometry and they rendered the scattering effect with a parametric filter structure. Fan et al [59] evaluated sound propagation effects such as scattering and occlusion from objects by using CNN.…”

Fast Acoustic Scattering Using Convolutional Neural Networks

“…Pulkki and Svensson [16] trained a small fully-connected neural network to learn exterior scattering from rectangular plates as predicted by the Biot-Tolstoy-Medwin (BTM) diffraction model [17]. The input was a carefully designed low-dimensional representation of the geometric configuration of source, plate, and listener based on knowledge of diffraction physics.…”

“…We restrict the problem to convex shapes to rule out reverberation and resonance effects in this initial study. In contrast to [16], our goal is to design a neural network that generalizes well for a variety of input shapes by formulating the problem as high-dimensional image-to-image regression which allows application of state-of-the-art convolutional neural networks (CNNs) that have been successfully applied in a variety of computer vision tasks [18][19][20].…”

“…CNNs are particularly well-adapted to such tasks. Further, it allows us to train a single network unlike [16], where occluded and unoccluded cases had to be treated separately with distinct networks.…”

Learning Acoustic Scattering Fields for Dynamic Interactive Sound Propagation

“…Not only can the objects undergo arbitrary motion or deformation, but their topologies may also change. In addition to specular and diffuse effects, it is also important to simulate complex diffracted scattering, occlusions, and inter-reflections that are perceptible [17,31,33]. Prior geometric methods are accurate in terms of simulating high-frequency effects and can be augmented with approximate edge diffraction methods that may work well in certain cases [41,49], though their behavior can be erratic [36].…”