Geometric Multimodal Contrastive Representation Learning

Abstract

Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method comprised of two main components: i) a twolevel architecture consisting of modality-specific base encoder, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermediate representations to a latent representation space; ii) a multimodal contrastive loss function that encourages the geometric alignment of the learned representations. We experimentally demonstrate that GMC representations are semantically rich and achieve state-of-the-art performance with missing modality information on three different learning problems including prediction and reinforcement learning tasks.

Robust Contrastive Learning With Dynamic Mixed Margin

“…By adopting crossmodal attention modules, MulT successfully fuses the information of multiple modalities and achieves superior performance than standard early fusion and late fusion approaches. Poklukar et al [46] further enhance the learned representation of MulT with their Geometric Multimodal Contrastive (GMC) Loss (e.g., L CE + L GMC ) and accomplish new state-of-the-art performance when there are any missing modalities in observed instances. For DMM, all of the experimental environments are set the same with [46], and we just add our Dynamic Mixed Margin Loss on GMC with hyperparameter ξ (e.g., L CE + L GMC + ξ L DMM ) that control the relative magnitude between DMM and other loss terms.…”

“…In this paper, we propose a modified contrastive learning method, dynamic mixed margin (DMM), that is compatible with diverse previous works. To validate our method, we apply our methods on COOT [1] and other variants [46] for video-text retrieval and video classification task.…”

“…Recently, Poklukar et al [46] proposed a new contrastive learning objective called GMC to learn a multimodal representation that satisfies two criteria: informativeness and robustness. By inducing the strong alignment between multimodal joint representation and unimodal representations, it learned semantically rich embedding space and achieved state-of-the-art performance on video clip classification in missing modalities setting.…”

“…To measure the sentiment analysis performance from multimodal input data, following [46], we employ CMU-MOSEI [62] dataset. CMU-MOSEI contains 23K video clips from YouTube with image, text, and audio annotation.…”