SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval

Abstract

We tackle the cross-modal retrieval problem, where the training is only supervised by the relevant multi-modal pairs in the data. The contrastive learning is the most popular approach for this task. However, its sampling complexity for learning is quadratic in the number of training data points. Moreover, it makes potentially wrong assumption that the instances in different pairs are automatically irrelevant. To address these issues, we propose a novel loss function that is based on self-labeling of the unknown classes. Specifically, we aim to predict class labels of the data instances in each modality, and assign those labels to the corresponding instances in the other modality (i.e., swapping the pseudo labels). With these swapped labels, we learn the data embedding for each modality using the supervised cross-entropy loss, hence leading to linear sampling complexity. We also maintain the queues for storing the embeddings of the latest batches, for which clustering assignment and embedding learning are done at the same time in an online fashion. This removes computational overhead of injecting intermittent epochs of entire training data sweep for offline clustering. We tested our approach on several realworld cross-modal retrieval problems, including text-based video retrieval, sketch-based image retrieval, and imagetext retrieval, and for all these tasks our method achieves significant performance improvement over the contrastive learning.

Adversarial Contrastive Learning by Permuting Cluster Assignments

“…Caron et al [2020] propose SwAV, an image representation technique that instead of comparing representations of different image views, follows a swapped prediction strategy to predict the code of one view from another view of the same image by clustering image representations with a computationally efficient online clustering approach. Similar swapping strategies have been utilized in cross-modal retrieval Kim [2021]. SEER Goyal et al [2021] is trained on large-scale unconstrained and uncurated image collections by improving the scalability of SwAV in terms of GPU memory consumption and training speed.…”

Semantic Role Aware Correlation Transformer For Text To Video Retrieval

“…In addition to this semantic gap, Luo et al [18] use only self-attentions among all modalities, and their method is highly dependent on pretraining. Kim et al [19] use only soft-attentions and ignore interaction between visual features. While Yang et al [20] utilize a fine-grained alignment with an extra loss and exploit special arrangement for hard negatives, they still use only selfattentions among all modalities by ignoring the interaction between visual features.…”