Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations

Abstract

The enhancement of unsupervised learning of sentence representations has been significantly achieved by the utility of contrastive learning. This approach clusters the augmented positive instance with the anchor instance to create a desired embedding space. However, relying solely on the contrastive objective can result in sub-optimal outcomes due to its inability to differentiate subtle semantic variations between positive pairs. Specifically, common data augmentation techniques frequently introduce semantic distortion, leading to a semantic margin between the positive pair. While the InfoNCE loss function overlooks the semantic margin and prioritizes similarity maximization between positive pairs during training, leading to the insensitive semantic comprehension ability of the trained model. In this paper, we introduce a novel Identical and Fraternal Twins of Contrastive Learning (named IFTCL) framework, capable of simultaneously adapting to various positive pairs generated by different augmentation techniques. We propose a Twins Loss to preserve the innate margin during training and promote the potential of data enhancement in order to overcome the sub-optimal issue. We also present proof-of-concept experiments combined with the contrastive objective to prove the validity of the proposed Twins Loss. Furthermore, we propose a hippocampus queue mechanism to restore and reuse the negative instances without additional calculation, which further enhances the efficiency and performance of the IFCL. We verify the IFCL framework on nine semantic textual similarity tasks with both English and Chinese datasets, and the experimental results show that IFCL outperforms state-of-the-art methods.

Topic-DPR: Topic-based Prompts for Dense Passage Retrieval

“…Employing one prompt for all disciplines may result in a uniform semantic shift and a convergence of passage representations in a restricted region of the embedding space, as depicted in Figure 1. This semantic space collapse Gao et al, 2019;Xiao et al, 2023) can blur the distinction between relevant and irrelevant passages, potentially masking irrelevant passages amidst relevant ones. Therefore, prompt generation is pivotal for this semantically nuanced task.…”