Deep Convolutional Neural Networks and Data Augmentation for Acoustic Event Detection

Gygli, Michael; Pfister, Beat; Gool, Luc Van

doi:10.48550/arxiv.1604.07160

Cited by 31 publications

(24 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The early neural network-based method directly detects the sound events via fully connected layers [11]. With the development of deep learning in computer vision, the convolutional neural network architecture is adopted to obtain the suitable temporal frequency representations of audio [12], [13], [14], [15], [16]. Due to the sequential nature of audio, recurrent neural networks are also used to learn the long-time feature representation of audio [17].…”

Section: A Sound Event Detection In Audiomentioning

confidence: 99%

Query-graph with Cross-gating Attention Model for Text-to-Audio Grounding

Tang¹,

Wang²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we address the text-to-audio grounding issue, namely, grounding the segments of the sound event described by a natural language query in the untrimmed audio. This is a newly proposed but challenging audio-language task, since it requires to not only precisely localize all the on-and off-sets of the desired segments in the audio, but to perform comprehensive acoustic and linguistic understandings and reason the multimodal interactions between the audio and query. To tackle those problems, the existing method treats the query holistically as a single unit by a global query representation, which fails to highlight the keywords that contain rich semantics. Besides, this method has not fully exploited interactions between the query and audio. Moreover, since the audio and queries are arbitrary and variable in length, many meaningless parts of them are not filtered out in this method, which hinders the grounding of the desired segments.To this end, we propose a novel Query Graph with Cross-gating Attention (QGCA) model, which models the comprehensive relations between the words in query through a novel query graph. Besides, to capture the fine-grained interactions between audio and query, a cross-modal attention module that assigns higher weights to the keywords is introduced to generate the snippet-specific query representations. Finally, we also design a cross-gating module to emphasize the crucial parts as well as weaken the irrelevant ones in the audio and query. We extensively evaluate the proposed QGCA model on the public Audiogrounding dataset with significant improvements over several state-of-the-art methods. Moreover, further ablation study shows the consistent effectiveness of different modules in the proposed QGCA model.

show abstract

Section: A Sound Event Detection In Audiomentioning

confidence: 99%

Query-graph with Cross-gating Attention Model for Text-to-Audio Grounding

Tang¹,

Wang²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This framework maps audio bases, extracted by non-negative matrix factorization (NMF), to the detected visual objects. In recent year, audio event detection (AED) [8,29,36] has received attention in the research community. Most of the AED methods locate audio events and then classify each event.…”

Section: Related Workmentioning

confidence: 99%

Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event Captioning

Rahman

Sigal

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Multi-modal learning, particularly among imaging and linguistic modalities, has made amazing strides in many high-level fundamental visual understanding problems, ranging from language grounding to dense event captioning. However, much of the research has been limited to approaches that either do not take audio corresponding to video into account at all, or those that model the audiovisual correlations in service of sound or sound source localization. In this paper, we present the evidence, that audio signals can carry surprising amount of information when it comes to high-level visual-lingual tasks. Specifically, we focus on the problem of weakly-supervised dense event captioning in videos and show that audio on its own can nearly rival performance of a state-of-the-art visual model and, combined with video, can improve on the state-of-theart performance. Extensive experiments on the ActivityNet Captions dataset show that our proposed multi-modal approach outperforms state-of-the-art unimodal methods, as well as validate specific feature representation and architecture design choices.

show abstract

“…The most widely used deep learning technique for audio classification is to apply a 2D convolutional network on the spectrogram or other derived features [12]- [14]. However, to the best of our knowledge, no other deep learning scheme has specifically addressed classification/segmentation into all of the three classes -Music, Speech and noise under a single unified framework.…”

Section: A 2d Convolutional Neural Network For Distillation and Compa...mentioning

confidence: 99%

SwishNet: A Fast Convolutional Neural Network for Speech, Music and Noise Classification and Segmentation

Hussain¹,

Haque²

2018

Preprint

View full text Add to dashboard Cite

Speech, Music and Noise classification/segmentation is an important preprocessing step for audio processing/indexing. To this end, we propose a novel 1D Convolutional Neural Network (CNN) -SwishNet. It is a fast and lightweight architecture that operates on MFCC features which is suitable to be added to the front-end of an audio processing pipeline. We showed that the performance of our network can be improved by distilling knowledge from a 2D CNN, pretrained on ImageNet. We investigated the performance of our network on the MUSAN corpus -an openly available comprehensive collection of noise, music and speech samples, suitable for deep learning. The proposed network achieved high overall accuracy in clip (length of 0.5-2s) classification (>97% accuracy) and frame-wise segmentation (>93% accuracy) tasks with even higher accuracy (>99%) in speech/non-speech discrimination task. To verify the robustness of our model, we trained it on MUSAN and evaluated it on a different corpus -GTZAN and found good accuracy with very little fine-tuning. We also demonstrated that our model is fast on both CPU and GPU, consumes a low amount of memory and is suitable for implementation in embedded systems.

show abstract

Deep Convolutional Neural Networks and Data Augmentation for Acoustic Event Detection

Cited by 31 publications

References 28 publications

Query-graph with Cross-gating Attention Model for Text-to-Audio Grounding

Query-graph with Cross-gating Attention Model for Text-to-Audio Grounding

Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event Captioning

SwishNet: A Fast Convolutional Neural Network for Speech, Music and Noise Classification and Segmentation

Contact Info

Product

Resources

About