Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval

Abstract

Constructing a joint representation invariant across dierent modalities (e.g., video, language) is of signicant importance in many multimedia applications. While there are a number of recent successes in developing eective image-text retrieval methods by learning joint representations, the video-text retrieval task, however, has not been explored to its fullest extent. In this paper, we study how to eectively utilize available multimodal cues from videos for the cross-modal video-text retrieval task. Based on our analysis, we propose a novel framework that simultaneously utilizes multi-modal features (dierent visual characteristics, audio inputs, and text) by a fusion strategy for ecient retrieval. Furthermore, we explore several loss functions in training the embedding and propose a modied pairwise ranking loss for the task. Experiments on MSVD and MSR-VTT datasets demonstrate that our method achieves signicant performance gain compared to the state-of-the-art approaches. CCS CONCEPTS • Information systems → Multimedia and multimodal retrieval; Learning to rank;

Dual Encoding for Zero-Example Video Retrieval

“…With big advances of deep learning in natural language processing and computer vision research, we observe an increased use of such techniques for video retrieval [7,24,34,36,37]. By directly encoding videos and text into a common space, these methods are concept free.…”

“…Though our goal is zeroexample video retrieval, which corresponds to text-to-video retrieval in the table, video-to-text retrieval is also included for completeness. While [7] is less effective than [24], letting the former use the same loss function as the latter brings in a considerable performance gain, with the sum of recalls increased from 90.3 to 132.1. The result suggests the importance of assessing different video / text encoding strategies within the same common space learning framework.…”

Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval

“…Each epoch training is just performed using a single GPU and takes no more than 10 minutes. [30] , MEE [27], MMEN [43], and JPoSE [43], and (3) other methods: JSFusion [49], CCA (FV HGLMM) [16], and Miech et al [26]. The experimental results on MSR-VTT and LSMDC are summarized, respectively, in Table 1 and Table 2.…”

Characterization and classification of semantic image-text relations

“…Numerous publications in recent years deal with multimodal information in retrieval tasks. The general problem of reduc-ing or bridging the semantic gap [44] between images and text is the main issue in cross-media retrieval [3,34,35,39,50]. Fan et al [8] tackle this problem by modeling humans' visual and descriptive senses with a multi-sensory fusion network.…”