Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Abstract

Learning common subspace is prevalent way to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared when achieved in cross-modal retrieval. Previous cross-modal retrieval methods focus on projecting the data between modalities into a common latent subspace by learning the correlation between them to bridge the modality gap. However, due to the rich semantic information in the video, the heterogeneous nature of audio-visual data leads to heterogeneous gaps in the distribution of modalities, which are difficult to be bridged into a common subspace by the previous cross-modality learning methods, which may lead to the loss of important semantic content in the video when the modality gap is eliminated by supervised learning, e.g., optimizing pairwise correlation, while the semantics of the categories may undermine the properties of the original features In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.