An Audio-Visual Dataset and Deep Learning Frameworks for Crowded Scene Classification

Abstract

In this paper, we propose a deep learning based system for the task of deepfake audio detection. In particular, the draw input audio is first transformed into various spectrograms using three transformation methods of Shorttime Fourier Transform (STFT), Constant-Q Transform (CQT), Wavelet Transform (WT) combined with different auditorybased filters of Mel, Gammatone, linear filters (LF), and discrete cosine transform (DCT). Given the spectrograms, we evaluate a wide range of classification models based on three deep learning approaches. The first approach is to train directly the spectrograms using our proposed baseline models of CNN-based model (CNN-baseline), RNN-based model (RNN-baseline), C-RNN model (C-RNN baseline). Meanwhile, the second approach is transfer learning from computer vision models such as ResNet-18, MobileNet-V3, EfficientNet-B0, DenseNet-121, SuffleNet-V2, Swint, Convnext-Tiny, GoogLeNet, MNASsnet, RegNet. In the third approach, we leverage the state-of-the-art audio pre-trained models of Whisper, Seamless, Speechbrain, and Pyannote to extract audio embeddings from the input spectrograms. Then, the audio embeddings are explored by a Multilayer perceptron (MLP) model to detect the fake or real audio samples. Finally, high-performance deep learning models from these approaches are fused to achieve the best performance. We evaluated our proposed models on ASVspoof 2019 benchmark dataset. Our best ensemble model achieved an Equal Error Rate (EER) of 0.03, which is highly competitive to top-performing systems in the ASVspoofing 2019 challenge.Experimental results also highlight the potential of selective spectrograms and deep learning approaches to enhance the task of audio deepfake detection.Items-deepfake audio, deep learning model, spectrogram, ASVspoof dataset.

Robust, General, and Low Complexity Acoustic Scene Classification Systems and An Effective Visualization for Presenting a Sound Scene Context

“…Rather than selecting a single task, we evaluate over a wide range of datasets of: Crowded-Scene [32], DCASE 2018 Task 1A and 1B [33], DCASE 2019 Task 1A and 1B [34], DCASE 2020 Task1A [35], DCASE 2021 Task 1A [35], and DCASE 2022 Tasks 1 [35]. We will see that the performance of our proposed system is competitive with the state-of-theart systems.…”

“…Crowded Scenes (Cr-Sc) [32]: contains 341 videos collected from YouTube (in-the-wild scenes), which presents a total recording time of nearly 29.06 hours. These videos were then split into 10-second video segments, each of which was annotated by one of five categories: 'Riot', 'Noise-Street', 'Firework-Event', 'Music-Event', or 'Sport-Atmosphere'.…”

“…Notably, 10-second video segments split from an original video are not presented in both Train and Test subsets to make the data distribution different between these two subsets. In this paper, we extract audio recordings from these video segments and follow the splitting method as proposed in [32] to evaluate this dataset.…”