Knowledge Distillation in Acoustic Scene Classification

Abstract

Common acoustic properties that different classes share degrades the performance of acoustic scene classification systems. This results in a phenomenon where a few confusing pairs of acoustic scenes dominate a significant proportion of all misclassified audio segments. In this paper, we propose adopting a knowledge distillation framework that trains deep neural networks using soft labels. Soft labels, extracted from another pre-trained deep neural network, are used to reflect the similarity between different classes that share similar acoustic properties. We also propose utilizing specialist models to provide additional soft labels. Each specialist model in this study refers to a deep neural network that concentrates on discriminating a single pair of acoustic scenes that are frequently misclassified. Self multi-head attention is explored for training specialist deep neural networks to further concentrate on target pairs of classes. The goal of this paper is to train a single deep neural network that demonstrates performance equivalent to, or higher than, an ensemble of multiple models, by distilling the knowledge from several models. Diverse experiments conducted using the detection and classification of acoustic scenes and events 2019 task 1-a dataset demonstrate that the knowledge distillation framework is effective in acoustic scene classification. Specialist models successfully decrease the number of misclassified audio segments in the target classes. The final single model with the proposed method that is trained by the proposed knowledge distillation from several models, including specialists trained using an attention mechanism, shows a classification accuracy of 77.63 %, higher than an ensemble of the baseline and multiple specialists.

Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech

“…Previous studies have already shown that deep convolutional neural networks trained to classify words, musical genres (Kell et al, 2018;Kumar et al, 2020) or natural sounds (Koumura et al, 2019), generate brain-like representations, in the sense that one can find a linear correspondence between the activation of the neural networks and the activations of the brain (Figure 1). This similarity can be quantified with a "brain score" (Jung et al, 2019), a correlation between the brain measurements and a linear projection of the model's activations, under the assumption that representations are defined as linearly exploitable information (Hung et al, 2005;Kamitani & Tong, 2005;Kriegeskorte & Kievit, 2013;King et al, 2018). We thus hypothesize that the nature of speech representations in the brain can be elucidated by comparing them to those of random, sound-generic and speech-specific neural networks (Figure 1).…”

DCASENET: An Integrated Pretrained Deep Neural Network for Detecting and Classifying Acoustic Scenes and Events

“…ASC is a multiclass classification task that identifies a segment as being one of predefined scenes (i.e., classes). Acoustic scenes have an abstract (i.e., ambiguous) definition, and thus various characteristics may coincide across different scenes [20,21]. For example, 'airport' and 'shopping mall', which are both predefined scenes in the DCASE ASC challenge, may contain people talking and the acoustic properties of large indoor spaces.…”

Lightweight Dual-Stream Residual Network for Single Image Super-Resolution

“…Network pruning reduces the size of the network by deleting some unnecessary branches [33], [34], but it still requires training a large network and designing methods to evaluate the importance of branches. Knowledge distillation requires a large teacher network to guide smaller distilled network training [35], [36]. It also needs to calculate the gap between the two network feature maps, which requires more complex techniques.…”