Self-Training for Sound Event Detection in Audio Mixtures

Abstract

Sound event detection (SED) takes on the task of identifying presence of specific sound events in a complex audio recording. SED has tremendous implications in video analytics, smart speaker algorithms and audio tagging. Recent advances in deep learning have afforded remarkable advances in performance of SED systems; albeit at the cost of extensive labeling efforts to train supervised methods using fully described sound class labels and timestamps. In order to address limitations in availability of training data, this work proposes a self-training technique to leverage unlabeled datasets in supervised learning using pseudo label estimation. This approach proposes a dual-term objective function: a classification loss for the original labels and expectation loss for pseudo labels. The proposed self training technique is applied to sound event detection in the context of the DCASE 2020 challenge, and reports a notable improvement over the baseline system for this task. The self-training approach is particularly effective in extending the labeled database with concurrent sound events.

Cross-Referencing Self-Training Network for Sound Event Detection in Audio Mixtures

“…In a previous work, a probabilistic expectation of potential labels was proposed as a pseudo label for unlabeled data [13]. In order to mitigate the issue of erroneous label mapping shown in Fig.…”

“…In order to reduce a computational load in pseudo label estimation, in this study, the number of concurrent events, k, was considered up to 2 (K = 2) so that total 56 potential labels (= 1 + 10 + 45 for k = 0, k = 1, and k = 2, respectively) were used to estimate pseudo label. According to the previous study in SRST model [13], the class averaging f-score was saturated at K = 2 because the case, which three or more target sounds happen at a time, is unusual in practical environments. With both test datasets, sound intervals that are overlaid with other target sound were counted and the results are summarized in TABLE III.…”

“…With this background, the self-training with pseudo label for the best class was applied to object detection and hyperspectral image classification [27], [28]. Instead of picking up the best class, a pseudo label was estimated based on probabilistic expectation of potential labels due to the multiple target scenario for SED task [13]. Recently, Wei, et al studied a theoretical background of the empirical successes in self-training with deep learning [12].…”

“…The estimation of a pseudo label in every frame for all unlabeled data introduces a heavy computational load in the calculation for all potential labels. To reduce this computation, the number of concurrent events k is considered up to 2 based on previous work [13]. The probabilities for multi-sound labels are calculated using a dynamic programming technique (9).…”

“…As such, the accuracy of this pseudo label has important implications on the network's performance. In an earlier work, a method of pseudo label estimation for unlabeled data and a reliability of the pseudo label were proposed [13]. The pseudo label was estimated by a probabilistic expectation of all potential labels as these probabilities were calculated based on the Bernoulli process with posterior probabilities of each class.…”