PILOT: Introducing Transformers for Probabilistic Sound Event Localization

Abstract

Sound event localization aims at estimating the positions of sound sources in the environment with respect to an acoustic receiver (e.g. a microphone array). Recent advances in this domain most prominently focused on utilizing deep recurrent neural networks. Inspired by the success of transformer architectures as a suitable alternative to classical recurrent neural networks, this paper introduces a novel transformer-based sound event localization framework, where temporal dependencies in the received multi-channel audio signals are captured via self-attention mechanisms. Additionally, the estimated sound event positions are represented as multivariate Gaussian variables, yielding an additional notion of uncertainty, which many previously proposed deep learning-based systems designed for this application do not provide. The framework is evaluated on three publicly available multi-source sound event localization datasets and compared against state-of-the-art methods in terms of localization error and event detection accuracy. It outperforms all competing systems on all datasets with statistical significant differences in performance.

A Survey of Sound Source Localization with Deep Learning Methods

“…For example, the authors of [186] placed an 8-head attention layer after a series of convolutional layers to track the source location predictions over time for different sources (up to two sources in their experiments). In [187], Schymura et al used three 4-head self-attention encoders along the time axis after a series of convolutional layers before estimating the activity and location of several sound events (see Fig. 6).…”

“…Several systems consider only the magnitude spectrograms, such as [52], [140], [199], [204], while other consider only the phase spectrogram [128], [203] When considering both magnitude and phase, they can be stacked also in a third dimension (as well as channels). This representation has been employed in many neural-based SSL systems [41], [70], [131], [143], [147], [148], [152], [153], [187]. Other systems proposed to decompose the complexvalued spectrograms into real and imaginary parts [42], [119], [192], [205].…”

“…Fig.6. The self-attention-based neural network architecture of[187]. This system is made of a feature extraction bloc including convolutional layers (not detailed in the figure), followed by a self-attention module identical to a Transformer encoder.…”

Multiple Sound Source Localization in Three Dimensions Using Convolutional Neural Networks and Clustering Based Post-Processing

“…Various input features were proposed to be used with ANN-based sound source localization methods, such as interaural level, phase or time difference [16], [18], [51], [52], phase transform-based features [38], [53]- [56], magnitude and phase spectrograms of array signals [35], [41], [57]- [59] and even unprocessed audio waveforms [20], [60]- [66].…”