Time–Frequency Feature Fusion for Noise Robust Audio Event Classification

Abstract

This paper explores the use of three different two-dimensional time-frequency features for audio event classification with deep neural network back-end classifiers. The evaluations use spectrogram, cochleogram and constant-Q transform-based images for classification of 50 classes of audio events in varying levels of acoustic background noise, revealing interesting performance patterns with respect to noise level, feature image type and classifier. Evidence is obtained that two well-performing features, the spectrogram and cochleogram, make use of information that is potentially complementary in the input features. Feature fusion is thus explored for each pair of features, as well as for all tested features. Results indicate that a fusion of spectrogram and cochleogram information is particularly beneficial, yielding an impressive 50-class accuracy of over 96% in 0 dB SNR and exceeding 99% accuracy in 10 dB SNR and above. Meanwhile, the cochleogram image feature is found to perform well in extreme noise cases of − 5 dB and − 10 dB SNR.

Robust acoustic scene classification using a multi-spectrogram encoder-decoder framework

“…We conclude from this that the three spectrograms represent sounds in ways that have affinity for certain types of sounds (mirroring a conclusion in [37], albeit on very different types of sound data). It is therefore unsurprising that intelligently combining the three spectrograms into a high level feature vector can achieve significant performance gain over single spectrograms.…”

Robust Deep Learning Frameworks For Acoustic Scene and Respiratory Sound Classification

“…better for classes containing vehicular sounds (with the exception of the Metro class). It can be concluded that the three spectrograms represent sounds in ways that have affinity for certain types of sounds (mirroring a conclusion in [119],…”

Benchmarking Audio Signal Representation Techniques for Classification with Convolutional Neural Networks

“…In addition, gammatone filters, which model the human auditory system, are used for forming the time-frequency representation of audio signals [ 41 , 42 ], called gammatone-spectrogram or cochleagram. Constant- Q transform (CQT) [ 43 ] is another technique for frequency transformation of signal and this is used in time-frequency representation of audio signals [ 44 , 45 ].…”

“…As such, it might be possible to improve CNN using various signal representations. A number of strategies have been proposed to combine the learning from multiple representations [ 24 , 45 , 55 , 56 ]. Broadly, the methods can be categorized as early-fusion, mid-fusion, and late-fusion [ 57 , 58 , 59 , 60 ].…”