Sound Event Detection Transformer: An Event-based End-to-End Model for Sound Event Detection

Abstract

Sound event detection (SED) has gained increasing attention with its wide application in surveillance, video indexing, etc. Existing models in SED mainly generate frame-level predictions, converting it into a sequence multi-label classification problem, which inevitably brings a trade-off between event boundary detection and audio tagging when using weakly labeled data to train the model. Besides, it needs post-processing and cannot be trained in an end-to-end way. This paper firstly presents the 1D Detection Transformer (1D-DETR), inspired by Detection Transformer. Furthermore, given the characteristics of SED, the audio query and a one-to-many matching strategy for fine-tuning the model are added to 1D-DETR to form the model of Sound Event Detection Transformer (SEDT), which generates event-level predictions, end-to-end detection. Experiments are conducted on the URBAN-SED dataset and the DCASE2019 Task4 dataset, and both experiments have achieved competitive results compared with SOTA models. The application of SEDT on SED shows that it can be used as a framework for onedimensional signal detection and may be extended to other similar tasks.

SP-SEDT: Self-supervised Pre-training for Sound Event Detection Transformer

“…The proposed SP-SEDT is pre-trained on the DCASE2018 task5 development dataset (72984 clips), and the unlabeled subset (14412 clips) of the DCASE2019 task4 dataset. The weakly-labeled subset and the synthetic subset with strong labels (2045 clips) of the DCASE2019 task4 dataset are used to fine-tune the pre-trained model with the experiment setup of [7]. The DCASE2019 task4 dataset is the same with the DCASE2021 task4 dataset except for the synthetic subset.…”

“…Event-level loss: The matching between target and predicted events can be obtained by the Hungarian algorithm and oneto-many strategy [7]. For each event, the loss function includes location loss and classification loss:…”

“…The inconsistency between model training and evaluating casts a restriction on the effectiveness of the models. In our earlier work [7], to overcome the shortcomings of frame-based models, the object-based prediction mechanism of Detection Transformer (DETR) [8] in the field of image object detection has been borrowed to build an event-based model for SED with transformer (SEDT), which outputs event-level predictions directly and achieves competitive performance. The event-based model pursues the same goal with the detection task, namely the best event-level predictions.…”

A Systematic Review of Rare Events Detection Across Modalities using Machine Learning and Deep Learning

“…Low-level feature extraction involves generating basic features and cross-scale path embedding to enhance fine-grained details in video frame interpolation [27]. These encompass color features (such as histograms, color moments, and color spaces like RGB, HSV, & LAB), temporal features (such as texture and STFT), shape features (such as object trajectory and silhouette), and motion features (such as vectors, frame differencing, and optical flow) [28].…”

DiffSED: Sound Event Detection with Denoising Diffusion

“…They are often heavily manually designed with plenty of heuristics and data-specific parameter optimization, hence less scalable and reliable across different audio data. Event-level approaches, on the other hand, directly model the temporal boundaries of sound events, taking into account the correlation between frames, thereby eliminating the mundane post-processing step and are more generalizable (Ye et al 2021). In both approaches, existing methods rely on proposal prediction by regressing the start and end times of each, i.e., discriminative learning based.…”