Particle-filter tracking of sounds for frequency-independent 3D audio rendering from distributed B-format recordings

Abstract

Six-Degree-of-Freedom (6DoF) audio rendering interactively synthesizes spatial audio signals for a variable listener perspective based on surround recordings taken at multiple perspectives distributed across the listening area in the acoustic scene. Methods that rely on recording-implicit directional information and interpolate the listener perspective without the attempt of localizing and extracting sounds often yield high audio quality, but are limited in spatial definition. Methods that perform sound localization, extraction, and rendering typically operate in the time-frequency domain and risk introducing artifacts such as musical noise. We propose to take advantage of the rich spatial information recorded in the broadband time-domain signals of the multitude of distributed first-order (B-format) recording perspectives. Broadband time-variant signal extraction retrieving direct signals and leaving residuals to approximate diffuse and spacious sounds is less of a quality risk, and likewise is the broadband re-encoding to enhance spatial definition of both signal types. To detect and track direct sound objects in this process, we combine the directional data recorded at the single perspectives into a volumetric multi-perspective activity map for particle-filter tracking. Our technical and perceptual evaluation confirms that this kind of processing enhances the otherwise limited spatial definition of direct-sound objects of other broadband but signal-independent virtual loudspeaker object (VLO) or Vector-Based Intensity Panning (VBIP) interpolation approaches.

Object-Based Six-Degrees-of-Freedom Rendering of Sound Scenes Captured with Multiple Ambisonic Receivers

“…The system also separately renders the residual ambient components, which represent the Ambisonic receiver signals after the source objects have been subtracted from them. Therefore, in essence, the proposed system may be viewed as a natural multireceiver extension to the Coding and Multi-Parameterisation of Ambisonic Sound Scenes (COMPASS) single-receiver method [7] and is also similar to the approach proposed recently in [8]. With an emphasis on developing a practical system, the proposed processing approach is implemented as a real-time Virtual Studio Technology (VST) audio plug-in 1 .…”

“…The closest work to the method proposed in the present article is described in [8], which operated in the timedomain and combined: building 2D planar activity maps based on broadband grid-scanning methods from each receiver, followed by peak-finding to ascertain source position estimates; subsequent particle-filtering based tracking of active sound objects; and then the application of broadband beamforming and spatialization of the objects, mixed with ambient rendering. Building on the work of [8], the proposed system instead operates in the time-frequency domain and lends particular emphasis on real-time operation. It forgoes the use of computationally expensive activitymap-based source position estimation in favor of continuous DoA estimation methods followed by computing the intersecting points between receivers.…”

“…Evaluations involving actual listening tests were conducted for the systems described in [26,27,31,16,20,9,21,42,24,8]. The mean opinion scores for the multipoint source separation method of [27] indicated robust spatial reproduction compared with a stereo reference but lower naturalness compared with a mono reference.…”

“…The work compares mixed and sparse sound-field decomposition approaches against conventional plane-wave nonsparse alternatives, with demonstrably better results when using the former. Finally, listening tests of the system described in [8] were conducted using either fixed listener positions or a dynamic listener following a predefined linear trajectory. It was demonstrated that the system outperformed nonparametric interpolation based alternatives in the majority of cases.…”

Research on Audio Recognition Based on the Deep Neural Network in Music Teaching

“…is function can map the data of the test set to one of the given categories, thus realizing the category prediction of unknown data. At present, common classi ers include decision tree, logistic regression, support vector machine (SVM), Naive Bayes, k-nearest neighbor algorithm (KNN), BP neural network, and deep learning [11][12][13].…”