Spatial audio signal processing for binaural reproduction of recorded acoustic scenes – review and challenges

Abstract

Spatial audio has been studied for several decades, but has seen much renewed interest recently due to advances in both software and hardware for capture and playback, and the emergence of applications such as virtual reality and augmented reality. This renewed interest has led to the investment of increasing efforts in developing signal processing algorithms for spatial audio, both for capture and for playback. In particular, due to the popularity of headphones and earphones, many spatial audio signal processing methods have dealt with binaural reproduction based on headphone listening. Among these new developments, processing spatial audio signals recorded in real environments using microphone arrays plays an important role. Following this emerging activity, this paper aims to provide a scientific review of recent developments and an outlook for future challenges. This review also proposes a generalized framework for describing spatial audio signal processing for the binaural reproduction of recorded sound. This framework helps to understand the collective progress of the research community, and to identify gaps for future research. It is composed of five main blocks, namely: the acoustic scene, recording, processing, reproduction, and perception and evaluation. First, each block is briefly presented, and then, a comprehensive review of the processing block is provided. This includes topics from simple binaural recording to Ambisonics and perceptually motivated approaches, which focus on careful array configuration and design. Beamforming and parametric-based processing afford more flexible designs and shift the focus to processing and modeling of the sound field. Then, emerging machine- and deep-learning approaches, which take a further step towards flexibility in design, are described. Finally, specific methods for signal transformations such as rotation, translation and enhancement, enabling additional flexibility in reproduction and improvement in the quality of the binaural signal, are presented. The review concludes by highlighting directions for future research.

A Conceptual Framework for Immersive Acoustic Auralisation: Investigating the Key Attributes

“…The following Table 1 depicts the sound sources in several acoustic virtual reality research domains. A variety of devices have been demonstrated to be effective for spatial sound capture [40]. Jeon et al [37] employed binaural recording techniques with microphone arrays and portable sound level metres to capture environmental noise experienced by inhabitants in the actual indoor environment to be simulated in a virtual world.…”

Acoustic Imaging With Circular Microphone Array: A New Approach for Sound Field Analysis

“…T HE problem of analyzing and reconstructing the sound field through acquisitions by multiple microphones has been widely studied in the literature. The ever-evolving trends in immersive multimedia communication have triggered a rapid evolution of computational acoustics, from the early solutions of spatial audio processing and rendering to the emerging paradigms of interactive extended reality [1]. Nowadays, indeed, sound field analysis and processing play a crucial role for the signal processing community which aims to develop new models and flexible representations to efficiently manipulate acoustic scenes.…”

Model-Matching Principle Applied to the Design of an Array-Based All-Neural Binaural Rendering System for Audio Telepresence

“…locations with near life-like audio quality. At the near-end, either headphones [5]- [7] or loudspeaker arrays [8]- [10] can be used as the rendering device for AT, where the idea behind the latter rendering approach is similar to the sound field reconstruction techniques such as Ambisonics [11,12] and Wave Field Synthesis (WFS) [13,14]. In this study, we will focus on the Binaural Audio Telepresence (BAT).…”