Audio-based Near-Duplicate Video Retrieval with Audio Similarity Learning

Abstract

In this work, we address the problem of audio-based near-duplicate video retrieval. We propose the Audio Similarity Learning (AuSiL) approach that effectively captures temporal patterns of audio similarity between video pairs. For the robust similarity calculation between two videos, we first extract representative audio-based video descriptors by leveraging transfer learning based on a Convolutional Neural Network (CNN) trained on a large scale dataset of audio events, and then we calculate the similarity matrix derived from the pairwise similarity of these descriptors. The similarity matrix is subsequently fed to a CNN network that captures the temporal structures existing within its content. We train our network following a triplet generation process and optimizing the triplet loss function. To evaluate the effectiveness of the proposed approach, we have manually annotated two publicly available video datasets based on the audio duplicity between their videos. The proposed approach achieves very competitive results compared to three state-ofthe-art methods. Also, unlike the competing methods, it is very robust to the retrieval of audio duplicates generated with speed transformations.

Audio Retrieval with Natural Language Queries

“…More recently, [4] use a siamese neural network framework to learn to encode semantically similar audio close together in the embedding space. [35] address multimedia event detection using only audio data, while [36] tackle near-duplicate video retrieval by audio retrieval. These are purely audio-based methods that are applied to video datasets, but without using visual information.…”

Audio Retrieval with Natural Language Queries

“…More recently, [4] use a siamese neural network framework to learn to encode semantically similar audio close together in the embedding space. [32] address multimedia event detection using only audio data, while [33] tackle near-duplicate video retrieval by audio retrieval. These are purely audio-based methods that are applied to video datasets, but without using visual information.…”

AI-empowered promotional video generation

“…After training, the learned metric shall return small distances among data from the same class and large distances among data from di↵erent classes. DML plays important roles in a diverse set of applications including image/video retrieval [188][189][190][191][192], person re-identification 42 3.1. Overview [193][194][195][196][197], vehicle re-identification [198,199], and self-supervised learning [200][201][202][203].…”