EclipSE: Efficient Long-Range Video Retrieval Using Sight and Sound

Lin, Yan-Bo; Lei, Jie; Bansal, Mohit; Bertasius, Gedas

doi:10.1007/978-3-031-19830-4_24

Cited by 17 publications

(3 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With large scale and realistic datasets, e.g., AudioSet [21], advanced network architectures have been adopted for audio classification including convolutional neural networks [23,24], convolutional-attention networks [25,26], and recent pure-attention based networks [1,27]. Particularly, AST [1] outperforms previous state-of-the-art audio classification approaches, and obtains widely adoption in many tasks, e.g., multimodal event classification [28,29] and video retrieval [30]. CMKD [31] further designs cross-modal knowledge distillation between convolutional networks and AST for audio classification.…”

Section: Related Workmentioning

confidence: 99%

Multiscale Audio Spectrogram Transformer for Efficient Audio Classification

Zhu¹,

Omar²

2023

Preprint

View full text Add to dashboard Cite

Audio event has a hierarchical architecture in both time and frequency and can be grouped together to construct more abstract semantic audio classes. In this work, we develop a multiscale audio spectrogram Transformer (MAST) that employs hierarchical representation learning for efficient audio classification. Specifically, MAST employs one-dimensional (and two-dimensional) pooling operators along the time (and frequency domains) in different stages, and progressively reduces the number of tokens and increases the feature dimensions. MAST significantly outperforms AST [1] by 22.2%, 4.4% and 4.7% on Kinetics-Sounds, Epic-Kitchens-100 and VGGSound in terms of the top-1 accuracy without external training data. On the downloaded AudioSet dataset, which has over 20% missing audios, MAST also achieves slightly better accuracy than AST. In addition, MAST is 5× more efficient in terms of multiply-accumulates (MACs) with 42% reduction in the number of parameters compared to AST. Through clustering metrics and visualizations, we demonstrate that the proposed MAST can learn semantically more separable feature representations from audio signals.

show abstract

Section: Related Workmentioning

confidence: 99%

Multiscale Audio Spectrogram Transformer for Efficient Audio Classification

Zhu¹,

Omar²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…[7] introduces structured multi-scale temporal decoder for self-attention. [18] suggests replacing the corresponding video with information-rich audio. They all work to reduce the computational effort of long video modeling to improve the performance.…”

Section: Related Workmentioning

confidence: 99%

Query-aware Long Video Localization and Relation Discrimination for Deep Video Understanding

Xu,

Wei,

2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

The surge in video and social media content underscores the need for a deeper understanding of multimedia data. Most of the existing mature video understanding techniques perform well with short formats and content that requires only shallow understanding, but do not perform well with long format videos that require deep understanding and reasoning. Deep Video Understanding (DVU) Challenge aims to push the boundaries of multimodal extraction, fusion, and analytics to address the problem of holistically analyzing long videos and extract useful knowledge to solve different types of queries. This paper introduces a query-aware method for long video localization and relation discrimination, leveraging an imagelanguage pretrained model. This model adeptly selects frames pertinent to queries, obviating the need for a complete movie-level knowledge graph. Our approach achieved first and fourth positions for two groups of movie-level queries. Sufficient experiments and final rankings demonstrate its effectiveness and robustness.

show abstract

“…As acoustic measurements are unaffected by illumination changes and occlusions, they are a reliable solution for round-the-clock 24 hours monitoring. In addition, acoustic data is typically more compact and cost-effective to process compared to raw video data [20], [21]. Cui et al [22] introduced an audio dataset AFFIA3K for FFIA consisting of 3000 labelled audio clips and demonstrated the practicality of audio-based FIFA.…”

mentioning

confidence: 99%

Fish Feeding Intensity Assessment in Aquaculture: A New Audio Dataset AFFIA3K and a Deep Learning Algorithm

Meng

Liu

Zhao

et al. 2022

2022 IEEE 32nd International Workshop on Machine Learning for Signal Processing (MLSP)

View full text Add to dashboard Cite

Fish feeding intensity assessment (FFIA) aims to evaluate the change of fish appetite during the feeding process, which is potentially useful in industrial aquaculture. Previous methods are mainly based on computer vision techniques. However, these methods are limited by water refraction and uneven illumination. In this paper, we introduce a new approach for FFIA using audio. We create a new audio dataset for FFIA, namely AFFIA3K, which contains 3000 labelled audio clips of different fish feeding intensity (None, Weak, Medium, Strong). We present a deep learning framework for FFIA, where the audio signal is first transformed into acoustic features, i.e. mel spectrogram, then a convolutional neural network (CNN)-based model is used to classify the fish feeding intensity. Experimental results show that our approach achieves an mAP of 0.74 on the test set of AFFIA3K, and considerably outperforms baseline systems. This indicates the potential of our proposed approach in aquaculture applications.

show abstract

EclipSE: Efficient Long-Range Video Retrieval Using Sight and Sound

Cited by 17 publications

References 56 publications

Multiscale Audio Spectrogram Transformer for Efficient Audio Classification

Multiscale Audio Spectrogram Transformer for Efficient Audio Classification

Query-aware Long Video Localization and Relation Discrimination for Deep Video Understanding

Fish Feeding Intensity Assessment in Aquaculture: A New Audio Dataset AFFIA3K and a Deep Learning Algorithm

Contact Info

Product

Resources

About