Robust Binaural Localization of a Target Sound Source by Combining Spectral Source Models and Deep Neural Networks

Abstract

Despite there being clear evidence for top-down (e.g., attentional) effects in biological spatial hearing, relatively few machine hearing systems exploit top-down model-based knowledge in sound localisation. This paper addresses this issue by proposing a novel framework for binaural sound localisation that combines model-based information about the spectral characteristics of sound sources and deep neural networks (DNNs). A target source model and a background source model are first estimated during a training phase using spectral features extracted from sound signals in isolation. When the identity of the background source is not available, a universal background model can be used. During testing, the source models are used jointly to explain the mixed observations and improve the localisation process by selectively weighting source azimuth posteriors output by a DNN-based localisation system. To address the possible mismatch between training and testing, a model adaptation process is further employed on-the-fly during testing, which adapts the background model parameters directly from the noisy observations in an iterative manner. The proposed system therefore combines model-based and data-driven information flow within a single computational framework. The evaluation task involved localisation of a target speech source in the presence of an interfering source and room reverberation. Our experiments show that by exploiting model-based information in this way, sound localisation performance can be improved substantially under various noisy and reverberant conditions.

What Can Computational Models Learn From Human Selective Attention? A Review From an Audiovisual Unimodal and Crossmodal Perspective

“…Besides, binaural models are apt to make use of the spatial localization information to address the Cocktail Party problem. For instance, Ma et al (2018) train DNNs to localize acoustic features in full 360 • azimuth angles. After the training phase, the binaural localization with spectral features is used as prior knowledge in the top-down modulation of the model.…”

Automatic Spatial Audio Scene Classification in Binaural Recordings of Music

“…According to Rumsey's spatial audio scene-based paradigm [5], complex spatial audio scenes could be described using perceptual attributes describing (1) individual audio sources, (2) ensembles of sources, and (3) acoustic environments. However, most of the models, mimicking spatial hearing in humans, are limited to the localization of individual audio sources [6][7][8][9][10], ignoring higher-level attributes describing complex spatial audio scenes.…”

“…Early machine-listening algorithms, inspired by spatial hearing in humans, could be characterized as single-source localization models since they were capable of localization of only one audio source at a time [11,12]. More recently, multiple-source localization models have been developed [6][7][8][9], which constitutes an important step towards quantification of higher-level attributes (e.g., ensemble width), ultimately leading to a holistic characterization of complex spatial audio scenes. While their reported accuracy is deemed to be good, their applicability is limited, as they often require a priori knowledge about the number of sources of interest and their signal characteristics.…”

“…While their reported accuracy is deemed to be good, their applicability is limited, as they often require a priori knowledge about the number of sources of interest and their signal characteristics. Another drawback of the multiple-source binaural localization models is that they were developed and tested predominately using speech signals [6][7][8][9]. Hence, their applicability to music recordings is unknown.…”

Binaural Sound Source Localization Based on Convolutional Neural Network

“…Later, Ma et al [Ma, May and Brown (2017)] used DNN and head rotation to achieve multi-sound source localization under reverberation conditions. Ma et al [Ma, Gonzalez, Brown et al (2018)] also combined sound spectrum and DNN in the time-frequency (TF) unit to estimate the azimuth. Yiwere et al [Yiwere and Rhee (2017)] used ILD and CCF as input features to train DNN models.…”