Spatial Audio Feature Discovery with Convolutional Neural Networks

Abstract

The advent of mixed reality consumer products brings about a pressing need to develop and improve spatial sound rendering techniques for a broad user base. Despite a large body of prior work, the precise nature and importance of various sound localization cues and how they should be personalized for an individual user to improve localization performance is still an open research problem. Here we propose training a convolutional neural network (CNN) to classify the elevation angle of spatially rendered sounds and employing Layerwise Relevance Propagation (LRP) on the trained CNN model. LRP provides saliency maps that can be used to identify spectral features used by the network for classification. These maps, in addition to the convolution filters learned by the CNN, are discussed in the context of listening tests reported in the literature. The proposed approach could potentially provide an avenue for future studies on modeling and personalization of head-related transfer functions (HRTFs).

CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings

“…Layerwise relevance propagation (LRP) [32] is a visualization technique which highlights the input features that are relevant for a given output. It can bring new information on the input features, for example in binaural localization where it has been used to identify the relevant elevation cues for a neural network, which can then be compared to human localization [33]. In addition, it is of paramount importance to check that the performance of the network is based on robust reasoning and not, for example, a bias in the dataset, which is made possible by LRP [34].…”

Blind Room Volume Estimation from Single-channel Noisy Speech

“…Furthermore, alternative feature representations, e.g., the echo density profile or the phase spectrum [28] could be studied. Neural network explanatory techniques [29] may allow to identify the most suitable spectro-temporal signal representation for blind volume estimation.…”

Binaural sound localization based on deep neural network and affinity propagation clustering in mismatched HRTF condition

“…While it works in the matched HRTF condition. In [20], a CNN-based sound localization method is proposed and proved to be robust to inter-subject and measurement variability, but this study only focuses on elevation localization. In [21], an end-to-end binaural sound localization approach is proposed, which estimates the azimuth directly from the waveform by CNN.…”