Data-Efficient Framework for Real-World Multiple Sound Source 2d Localization

Abstract

Deep neural networks have recently led to promising results for the task of multiple sound source localization. Yet, they require a lot of training data to cover a variety of acoustic conditions and microphone array layouts. One can leverage acoustic simulators to inexpensively generate labeled training data. However, models trained on synthetic data tend to perform poorly with real-world recordings due to the domain mismatch. Moreover, learning for different microphone array layouts makes the task more complicated due to the infinite number of possible layouts. We propose to use adversarial learning methods to close the gap between synthetic and real domains. Our novel ensemble-discrimination method significantly improves the localization performance without requiring any label from the real data. Furthermore, we propose a novel explicit transformation layer to be embedded in the localization architecture. It enables the model to be trained with data from specific microphone array layouts while generalizing well to unseen layouts during inference.

Direction of Arrival Estimation of Sound Sources Using Icosahedral CNNs

“…It seems that we might be approaching the accuracy limit imposed by this dataset difference, so the accuracy with real recordings can not improve even when we improve the models. In recent years, several domain adaptation techniques have been proposed to improve the accuracy of models trained with simulated signals [48], [49], [50], [51] and it would be interesting to conduct further studies along these lines.…”

A Survey of Sound Source Localization with Deep Learning Methods

“…The major drawback of the DNN-based approaches is the lack of generality. A deep model designed for and trained in a given configuration (for example a given microphone array geometry) will not provide satisfying localization results if the setup changes [24], [25], unless some relevant adaptation method can be used, which is still an open problem in deep learning in general. In this paper, we do not consider this aspect.…”

“…They evaluated several types of outputs (binary, Gaussian-based, and binary followed by regression refinement) which showed promising results on the simulated and real data. They extended their work in [25] where they proposed to use adversarial training (see Section VIII) to improve the network performance on real data, as well as on microphone arrays unseen in the training set, in an unsupervised training scheme. To do that, they introduced a novel explicit transformation layer which helps the network to be invariant to the microphone array layout.…”

“…2) Cartesian coordinates: A few works applied the classification paradigm to estimate the cartesian coordinates. In [25], [122], [192], the horizontal (x, y) plane is divided into small regions of same size, each of them being a class in the output layer. However this representation suffers from a decreasing angular difference between the regions that are far from the microphone array, which is probably why regression is usually preferred for estimating cartesian coordinates.…”

“…In [25], a semi-supervised approach is proposed to adapt the network to real-world data after it was trained with a simulated dataset. This strategy is implemented with adversarial training [251].…”