Temporal Cross-Media Retrieval with Soft-Smoothing

Abstract

Multimedia information have strong temporal correlations that shape the way modalities co-occur over time. In this paper we study the dynamic nature of multimedia and social-media information, where the temporal dimension emerges as a strong source of evidence for learning the temporal correlations across visual and textual modalities. So far, cross-media retrieval models, explored the correlations between different modalities (e.g. text and image) to learn a common subspace, in which semantically similar instances lie in the same neighbourhood. Building on such knowledge, we propose a novel temporal cross-media neural architecture, that departs from standard cross-media methods, by explicitly accounting for the temporal dimension through temporal subspace learning. The model is softly-constrained with temporal and inter-modality constraints that guide the new subspace learning task by favouring temporal correlations between semantically similar and temporally close instances. Experiments on three distinct datasets show that accounting for time turns out to be important for cross-media retrieval. Namely, the proposed method outperforms a set of baselines on the task of temporal cross-media retrieval, demonstrating its effectiveness for performing temporal subspace learning.

Diachronic Cross-modal Embeddings

“…In Uricchio et al [29], the value of temporal information for the tasks of image annotation and retrieval, such as tag frequency, is recognised. In order to model the temporal behaviour of data, embeddings must retain temporal correlations [2,9,15,24,27,38]. The challenge resides in capturing such correlations and incorporating them in cross-modal embeddings.…”

“…The focus has been on capturing correlations between instances (using category information when available), without accounting for temporal correlations, for data organisation. Recently, a temporal cross-modal common space approach was proposed [27], where temporal correlations are used to organise instances in a static embedding space. The authors observed that for dynamic data, the incorporation of temporal insights increases retrieval performance.…”

“…Building on the most recent state-of-the-art cross-modal learning works [21,27,30,33,37], we adopt the ranking loss function as the model base loss. In its general formulation, triplets (x a * , x p * , x n * ), are composed by an anchor element x a * , that should be more similar to positive elements x p * sharing a category, than to negative elements x n * not sharing categories, by at least a margin m. Triplet constraints are expressed as s(x a * , x p * ) > s(x a * , x n * )+m, and then turned into a differentiable function, by means of a relaxation under the hinge loss function [11]:…”

“…without preserving the time dimension, and thus ignoring the evolution of modality interactions [5,7,12,20,21,23,27,30,31,35,36]. Approaches have ranged from solutions that organize the space according to linear correlations [23,33,36] (image and texts cooccurrence), semantic [20,30,34,35] (category information) and/or temporal correlations [27]. State-of-the-art methods can preserve semantic and temporal correlations but do not consider the time dimension in its original scale, thus resulting in the loss of all the information regarding the evolution of data.…”

Cross-Modal Subspace Learning with Scheduled Adaptive Margin Constraints

“…Modality projections into cross-modal subspaces must capture both inter-category and inter-modality correlations in that subspace. To this end, the cross-modal subspace learning problem is commonly formulated using a maximum-margin learning approach, by imposing a set of constraints over pairwise instance's similarity, on the target subspace [17,22,25,28,34].…”