Learning to Hash Naturally Sorts

Abstract

Transformer-based architectures with grid features represent the state-of-the-art in visual and language reasoning tasks, such as visual question answering and image-text matching. However, directly applying them to image captioning may result in spatial and fine-grained semantic information loss. Their applicability to image captioning is still largely under-explored. Towards this goal, we propose a simple yet effective method, Spatial- and Scale-aware Transformer (S2 Transformer) for image captioning. Specifically, we firstly propose a Spatial-aware Pseudo-supervised (SP) module, which resorts to feature clustering to help preserve spatial information for grid features. Next, to maintain the model size and produce superior results, we build a simple weighted residual connection, named Scale-wise Reinforcement (SR) module, to simultaneously explore both low- and high-level encoded features with rich semantics. Extensive experiments on the MSCOCO benchmark demonstrate that our method achieves new state-of-art performance without bringing excessive parameters compared with the vanilla transformer. The source code is available at https://github.com/zchoi/S2-Transformer

Weighted Contrastive Hashing

“…The quality of the hash codes obtained in this way is not high, which will have an impact on the final retrieval results. This can also explain why NSH [36] boosts highly on single-label datasets, but the boost of MAP on multi-label datasets is not very obvious.…”

“…Earlier studies rely heavily on artificial annotations, which makes it difficult to apply in real-world scenarios due to the high labor costs. As a result, unsupervised deep hashing [27,22,23,36] has gradually become the major research direction in this field, with the recent boom in unsupervised learning [3,13,4,26,2,12]. The key difficulty with unsupervised hash is that the ad-hoc encoding process does not extract the key information for hashing, precisely because of the lack of supervised information.…”

“…which constructs similar and dissimilar instances and learns the discrete representations by prompting the model to pull in the similar instances and push away the dissimilar instances. With simply the most fundamental concepts for contrastive learning, existing methods [27,23,36] based on contrastive learning have achieved significant success. Despite their success, most of the current methods mainly focus on adjusting the contrastive loss to fit the hash learning criterion [27,23].…”

“…Curiously, most existing approaches do not focus on this problem. To the best of our knowledge, NSH [36] uses Neural Sorting Operators to obtain the permutation of a vector of similarity scores, and it employs the sorted similarity results to pick the top m positive samples of the anchor, i.e., it improves the comparison by increasing the number of positive samples in the learning framework. However, in the experiment, the optimal number of positive samples is fixed at 3, and all of them are considered fully similar.…”

“…To obtain the similar samples of an anchor, we use the aggregated vector of similarity between the patches of different samples as weights. Unlike NSH [36], we do not selectively pick the most relevant samples as the positive samples for contrastive learning, but assign trainable weights to all candidate samples, which represent the degree of similarity between samples. That is, we can consider an image pair as less or more similar, rather than stating them as fully similar or dissimilar in absolute terms.…”