Matching-oriented Embedding Quantization For Ad-hoc Retrieval

Abstract

Product quantization (PQ) is a widely used technique for ad-hoc retrieval. Recent studies propose supervised PQ, where the embedding and quantization models can be jointly trained with supervised learning. However, there is a lack of appropriate formulation of the joint training objective; thus, the improvements over previous non-supervised baselines are limited in reality. In this work, we propose the Matching-oriented Product Quantization (MoPQ), where a novel objective Multinoulli Contrastive Loss (MCL) is formulated. With the minimization of MCL, we are able to maximize the matching probability of query and ground-truth key, which contributes to the optimal retrieval accuracy. Given that the exact computation of MCL is intractable due to the demand of vast contrastive samples, we further propose the Differentiable Cross-device Sampling (DCS), which significantly augments the contrastive samples for precise approximation of MCL. We conduct extensive experimental studies on four realworld datasets, whose results verify the effectiveness of MoPQ. The code is available at https://github.com/microsoft/MoPQ. 1. We adopt the Asymmetric Distance Computation (ADC) (Jégou et al., 2011), where only keys need to be quantized.

Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings

“…One potential problem about the above works is that VQ is learned purely based on document embeddings, whereas the relationships with queries are not taken into consideration. Besides, it is also analyzed in [45] that the reduction of the reconstruction loss does not necessarily improve the retrieval quality. Realizing such defects, the latest works (e.g., MoPQ [45], JPQ [50], RepCONC [51]) propose to jointly learn VQ and the embedding models via contrastive learning, in which the discrimination of the ground-truth documents for each query can be optimized (referred as retrieval loss minimization), as Figure 1 (C).…”

“…Besides, it is also analyzed in [45] that the reduction of the reconstruction loss does not necessarily improve the retrieval quality. Realizing such defects, the latest works (e.g., MoPQ [45], JPQ [50], RepCONC [51]) propose to jointly learn VQ and the embedding models via contrastive learning, in which the discrimination of the ground-truth documents for each query can be optimized (referred as retrieval loss minimization), as Figure 1 (C). Compared with the previous minimization of the reconstruction loss, the retrieval loss minimization achieves substantial improvements in terms of the retrieval performance.…”

“…Knowing that brute-force linear scan is temporally infeasible, the embeddings need to be organized by ANN index in order to support real-world information retrieval. For the past decade, the vector quantization (VQ) techniques, e.g., Inverted File System (IVF) [1,2,19] and Product Quantization (PQ) [18,40,45], are widely applied, which enables ANN to be performed with competitive time and memory efficiency.…”

“…Unfortunately, such an operation is inconsistent with the goal of retrieving groundtruth documents for the query, which may cause severe loss of retrieval performance. In recent works [45,50], the dense embeddings and VQ are jointly learned to minimize the post-quantization retrieval loss. However, the existing methods typically rely on contrastive learning, which calls for a tremendous amount of queries labeled with ground-truth documents.…”

“…(B): reconstructed embeddings are end-to-end differentiable and learned for reconstruction loss minimization; e.g., DPQ [8]. (C): reconstructed embeddings are end-to-end differentiable and learned for retrieval loss minimization; e.g., MoPQ [45], JPQ [50], RepCONC [51]. (D): quantized embeddings are learned by knowledge distillation based on fixed document embeddings (Distill-VQ).…”