End-to-end Binaural Sound Localisation from the Raw Waveform

Vecchiotti, Paolo; Ma, Ning; Squartini, Stefano; Brown, Guy J.

doi:10.1109/icassp.2019.8683732

Cited by 57 publications

(49 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the proposed WM softmax loss is replaced with a regular softmax loss, the output of the proposed network will not be the DP-RTF feature anymore. Instead, the network will output the class of source direction, which is the same as many deep-classification-based sound source localization methods, such as [5,25]. With 5 • error tolerance, the MSE loss slightly outperforms the softmax loss in most cases, but the softmax loss performs relatively better with 0 • error tolerance.…”

Section: Resultsmentioning

confidence: 99%

Supervised Direct-Path Relative Transfer Function Learning for Binaural Sound Source Localization

Yang

Liu

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Direct-path relative transfer function (DP-RTF) refers to the ratio between the direct-path acoustic transfer functions of two channels. Though DP-RTF fully encodes the sound directional cues and serves as a reliable localization feature, it is often erroneously estimated in the presence of noise and reverberation. This paper proposes a supervised DP-RTF learning method with deep neural networks for robust binaural sound source localization. To exploit the complementarity of single-channel spectrogram and dual-channel difference information, we first recover the direct-path magnitude spectrogram from the contaminated one using a monaural enhancement network, and then predict the DP-RTF from the dual-channel (enhanced-) intensity and phase cues using a binaural enhancement network. In addition, a weighted-matching softmax training loss is designed to promote the predicted DP-RTFs to be concentrated for the same direction and separated for different directions. Finally, the direction of arrival (DOA) of source is estimated by matching the predicted DP-RTF with the ground truths of candidate directions. Experimental results show the superiority of our method for DOA estimation in the environments with various levels of noise and reverberation.

show abstract

Section: Resultsmentioning

confidence: 99%

Supervised Direct-Path Relative Transfer Function Learning for Binaural Sound Source Localization

Yang

Liu

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…It can be seen that localization accuracies between different HRTFs are different, and the accuracies on the diagonal line correspond to the matched HRTF condition and all reach to 100%. Besides the diagonal line, the localization accuracy at (21,12) is also 100%, which indicates that the similarity between the HRTF 12 and HRTF 21 is very high and the DNNs trained by them respectively can be substituted for each other.…”

Section: Localization Similarity Between Hrtfsmentioning

confidence: 97%

“…In [20], a CNN-based sound localization method is proposed and proved to be robust to inter-subject and measurement variability, but this study only focuses on elevation localization. In [21], an end-to-end binaural sound localization approach is proposed, which estimates the azimuth directly from the waveform by CNN. This approach is robust to the reverberate condition; however, the performance in the HRTF-mismatched condition is not studied.…”

Section: Introductionmentioning

confidence: 99%

Binaural sound localization based on deep neural network and affinity propagation clustering in mismatched HRTF condition

Wang

Qian

Xie

et al. 2020

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Binaural sound source localization is an important and widely used perceptually based method and it has been applied to machine learning studies by many researchers based on head-related transfer function (HRTF). Because the HRTF is closely related to human physiological structure, the HRTFs vary between individuals. Related machine learning studies to date tend to focus on binaural localization in reverberant or noisy environments, or in conditions with multiple simultaneously active sound sources. In contrast, mismatched HRTF condition, in which the HRTFs used to generate the training and test sets are different, is rarely studied. This mismatch leads to a degradation of localization performance. A basic solution to this problem is to introduce more data to improve generalization performance, which requires a lot. However, simply increasing the data volume will result in data-inefficiency. In this paper, we propose a data-efficient method based on deep neural network (DNN) and clustering to improve binaural localization performance in the mismatched HRTF condition. Firstly, we analyze the relationship between binaural cues and the sound source localization with a classification DNN. Different HRTFs are used to generate training and test sets, respectively. On this basis, we study the localization performance of DNN model trained by each training set on different test sets. The result shows that the localization performance of the same model on different test sets is different, while the localization performance of different models on the same test set may be similar. The result also shows a clustering trend. Secondly, different HRTFs are divided into several clusters. Finally, the corresponding HRTFs of each cluster center are selected to generate a new training set and to train a more generalized DNN model. The experimental results show that the proposed method achieves better generalization performance than the baseline methods in the mismatched HRTF condition and has almost equal performance to the DNN trained with a large number of HRTFs, which means the proposed method is data-efficient.

show abstract

“…Further, binaural detection is a highly specialised auditory function for which deficits have real-world consequences 25,26 . DNNs may offer the opportunity to bridge this gap between animal and human data, and as yet, the inner workings of DNNs constructed to handle binaural audio have scarcely been considered [27][28][29] .…”

Section: Introductionmentioning

confidence: 99%

Inferring the basis of binaural detection with a modified autoencoder

Smith

Sollini

Akeroyd

2021

Preprint

View full text Add to dashboard Cite

SUMMARYParallels have been reported between broad organization in the auditory system and optimized artificial neural networks1–3. It remains to be seen whether such promising analogies between the auditory system and deep learning models endure at other levels of description. Here, we examined whether artificial neural networks4,5 could offer a mechanistic account of human behavior in an auditory task. The chosen task promoted the use of binaural cues (across the ears) to help detect a signal in noise6,7. In the optimal network, we observed the emergence of specialized computations with prominent similarities to in vivo animal data8. Artificial neurons developed a sensitivity to temporal delays that increased hierarchically, and were widely distributed in preference (extending to delays beyond the range permitted by head width). Ensuing dynamics were consistent with a binaural cross-correlation mechanism9. Given the neural mechanisms of binaural detection in humans is contested9–13, these findings help to resolve this debate. Moreover, this is a primary demonstration that deep learning can infer tangible mechanisms underlying auditory perception.

show abstract

End-to-end Binaural Sound Localisation from the Raw Waveform

Cited by 57 publications

References 22 publications

Supervised Direct-Path Relative Transfer Function Learning for Binaural Sound Source Localization

Supervised Direct-Path Relative Transfer Function Learning for Binaural Sound Source Localization

Binaural sound localization based on deep neural network and affinity propagation clustering in mismatched HRTF condition

Inferring the basis of binaural detection with a modified autoencoder

Contact Info

Product

Resources

About