Learning Sound Localization Better from Semantically Similar Samples

Senocak, Arda; Ryu, Hyeonggon; Kim, Junsik; Kweon, In So

doi:10.1109/icassp43922.2022.9747867

Cited by 25 publications

(14 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, we train our model with 144k VGG Sound samples and test on the Flickr SoundNet test set. Comparing against [6,29,30], we significantly outperform all methods, as shown in Table 1, which shows our model is capable of generalizing well across datasets. We further investigate our method's robustness by testing on sound categories that are disjoint from what is seen during training.…”

Section: Methodsmentioning

confidence: 84%

“…Training Set cIoU 0.5 AUC cIoU Attention [28] Flickr 10k 0.436 0.449 CoarseToFine [25] 0.522 0.496 AVObject [1] 0.546 0.504 LVS * [6] 0.730 0.578 SSPL [30] 0.743 0.587 HTF (Ours) 0.860 0.634 Attention [28] Flickr 144k 0.660 0.558 DMC [19] 0.671 0.568 LVS * [6] 0.702 0.588 LVS † [6] 0.697 0.560 HardPos [29] 0.762 0.597 SSPL [30] 0 ilarly perform random cropping and horizontal flipping of the flow fields, which are performed consistently with image augmentations. For audio, we sample 3 seconds of the video at 16kHz and construct a log-scaled spectrogram using a bin size of 256, FFT window of 512 samples, and stride of 274 samples, resulting in a shape of 257 × 300.…”

Section: Methodsmentioning

confidence: 99%

“…Training Set cIoU 0.5 AUC cIoU Attention [28] VGGSound 10k 0.160 0.283 LVS * [6] 0.297 0.358 SSPL [30] 0.314 0.369 HTF (Ours) 0.393 0.398 Attention [28] VGGSound 144k 0.185 0.302 AVObject [1] 0.297 0.357 LVS * [6] 0.301 0.361 LVS † [6] 0.288 0.359 HardPos [29] 0.346 0.380 SSPL [30] 0.339 0.380 HTF (Ours) 0.394 0.400 the Flickr testing set, we achieved an improved performance of 11.7% cIoU and 4.7% AUC when trained on 10k Flickr samples and 10.6% cIoU and 2.9% AUC when trained with 144k Flickr samples. Similarly, on the VGG Sound Source testing set, we improve by 7.9% cIoU and 2.9% AUC when trained on 10k VGG Sound samples and 5.5% cIoU and 2.0% AUC when trained on 144k samples.…”

Section: Methodsmentioning

confidence: 99%

“…In this section, we compare our method against prior works [1,6,19,25,28,29,30] on standardized experiments for self-supervised visual sound source localization. Results of various training configurations are reported in Tables 1 and 2 for the Flickr SoundNet and VGG Sound Source testing datasets, respectively.…”

Section: Quantitative Evaluationmentioning

confidence: 99%

See 3 more Smart Citations

Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization

Fedorishin¹,

Mohan²,

Jawade³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning to localize the sound source in videos without explicit annotations is a novel area of audio-visual research. Existing work in this area focuses on creating attention maps to capture the correlation between the two modalities to localize the source of the sound. In a video, oftentimes, the objects exhibiting movement are the ones generating the sound. In this work, we capture this characteristic by modeling the optical flow in a video as a prior to better aid in localizing the sound source. We further demonstrate that the addition of flow-based attention substantially improves visual sound source localization. Finally, we benchmark our method on standard sound source localization datasets and achieve state-of-the-art performance on the Soundnet Flickr and VGG Sound Source datasets. Code: https://github.com/denfed/heartheflow.

show abstract

Section: Methodsmentioning

confidence: 84%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Quantitative Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization

Fedorishin¹,

Mohan²,

Jawade³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…1. Research shows [4,16,29,36] that these false negatives will lead to contradictory objectives and harm the representation learning.…”

Section: Indicates Equal Contributionmentioning

confidence: 99%

Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning

Sun¹,

Zhang²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Self-supervised audio-visual source localization aims to locate sound-source objects in video frames without extra annotations. Recent methods often approach this goal with the help of contrastive learning, which assumes only the audio and visual contents from the same video are positive samples for each other. However, this assumption would suffer from false negative samples in real-world training. For example, for an audio sample, treating the frames from the same audio class as negative samples may mislead the model and therefore harm the learned representations (e.g., the audio of a siren wailing may reasonably correspond to the ambulances in multiple images). Based on this observation, we propose a new learning strategy named False Negative Aware Contrastive (FNAC) to mitigate the problem of misleading the training with such false negative samples. Specifically, we utilize the intra-modal similarities to identify potentially similar samples and construct corresponding adjacency matrices to guide contrastive learning. Further, we propose to strengthen the role of true negative samples by explicitly leveraging the visual features of sound sources to facilitate the differentiation of authentic sounding source regions. FNAC achieves state-of-the-art performances on Flickr-SoundNet, VGG-Sound, and AVSBench, which demonstrates the effectiveness of our method in mitigating the false negative issue. The code is available at https://github.com/OpenNLPLab/FNAC_AVL.

show abstract

Localizing Visual Sounds the Easy Way

Morgado

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Audio-visual generalized zero-shot learning is a rapidly advancing domain that seeks to understand the intricate relations between audio and visual cues within videos. The overarching goal is to leverage insights from seen classes to identify instances from previously unseen ones. Prior approaches primarily utilized synchronized auto-encoders to reconstruct audio-visual attributes, which were informed by crossattention transformers and projected text embeddings. However, these methods fell short of effectively capturing the intricate relationship between cross-modal features and class-label embeddings inherent in pretrained language-aligned embeddings. To circumvent these bottlenecks, we introduce a simple yet effective framework for Easy Audio-Visual Generalized Zero-shot Learning, named EZ-AVGZL , that aligns audio-visual embeddings with transformed text representations. It utilizes a single supervised text audio-visual contrastive loss to learn an alignment between audio-visual and textual modalities, moving away from the conventional approach of reconstructing cross-modal features and text embeddings.Our key insight is that while class name embeddings are well aligned with language-based audio-visual features, they don't provide sufficient class separation to be useful for zero-shot learning. To address this, our method leverages differential optimization to transform class embeddings into a more discriminative space while preserving the semantic structure of language representations. We conduct extensive experiments on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL benchmarks. Our results demonstrate that our EZ-AVGZL achieves state-of-the-art performance in audio-visual generalized zero-shot learning.

show abstract

Learning Sound Localization Better from Semantically Similar Samples

Cited by 25 publications

References 25 publications

Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization

Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization

Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning

Localizing Visual Sounds the Easy Way

Contact Info

Product

Resources

About