Self-supervised Video Retrieval Transformer Network

Abstract

Content-based video retrieval aims to find videos from a large video database that are similar to or even nearduplicate of a given query video. It plays an important role in many video related applications, including copyright protection, recommendation, filtering and etc.. Video representation and similarity search algorithms are crucial to any video retrieval system. To derive effective video representation, most video retrieval systems require a large amount of manually annotated data for training, making it costly inefficient. In addition, most retrieval systems are based on frame-level features for video similarity searching, making it expensive both storage wise and search wise. We propose a novel video retrieval system, termed SVRTN, that effectively addresses the above shortcomings. It first applies self-supervised training to effectively learn video representation from unlabeled data to avoid the expensive cost of manual annotation. Then, it exploits transformer structure to aggregate frame-level features into clip-level to reduce both storage space and search complexity. It can learn the complementary and discriminative information from the interactions among clip frames, as well as acquire the frame permutation and missing invariant ability to support more flexible retrieval manners. Comprehensive experiments on two challenging video retrieval datasets, namely FIVR-200K and SVD, verify the effectiveness of our proposed SVRTN method, which achieves the best performance of video retrieval on accuracy and efficiency.

Video Similarity and Alignment Learning on Partial Video Copy Detection

“…For efficient video similarity measurement. Most retrieval methods have a straightforward motivation aggregating the local frame-level features into clip-level or even video-level representations, such as global vectors [15,25], hash codes [8,9,19], Bag-of-Words (BoW) [2,14,16], and video similarity is measured by distances of aggregated representations. However, the aggregated representations are too coarse to cover abundant fine-grained information and can't be used to partial segment localization.…”

“…However, the aggregated representations are too coarse to cover abundant fine-grained information and can't be used to partial segment localization. Therefore in this paper, we adopt frame-level representation of SVRTN [8] to implement frame encoding, keeping the fine-grained information for partial segment localization.…”

“…As for deletion, some consecutive frames are randomly picked and removed from original sequences. After temporal transformation, three types of spatial transformations similar to [8] are considered: photometric variation, geometric transformation and other editing operations. Each training video is randomly transformed twice to a pair of sequences namely anchor and positive.…”

“…For FE, we adopt the implementation of SVRTN 𝑓 [8] from which we extract the float features and the feature dimension 𝑊 of framelevel representations including 𝑈 ,𝑉 ,𝑓 𝜃 (𝑈 ) and 𝑓 𝜃 (𝑉 ) are set to 512. For SE, we use a 8-head transformer encoder layer with 1024 hidden size and no positional embedding is added to the input features.…”

“…For similarity measurement, existing methods generally based on frame-level representation, such as local descriptor based representation [26], color correlation [6], DCT coefficients [7] and CNN deep feature [1,8,11,15,18]. The above representations mainly focus on the spatial information and neglect the temporal information, which causes that the video similarity is easily biased towards spatial information of the key frames.…”