Query-aware locality-sensitive hashing for approximate nearest neighbor search

Abstract

Locality-Sensitive Hashing (LSH) and its variants are the well-known indexing schemes for the c-Approximate Nearest Neighbor (c-ANN) search problem in high-dimensional Euclidean space. Traditionally, LSH functions are constructed in a query-oblivious manner in the sense that buckets are partitioned before any query arrives. However, objects closer to a query may be partitioned into different buckets, which is undesirable. Due to the use of query-oblivious bucket partition, the state-of-the-art LSH schemes for external memory, namely C2LSH and LSB-Forest, only work with approximation ratio of integer c ≥ 2. In this paper, we introduce a novel concept of query-aware bucket partition which uses a given query as the "anchor" for bucket partition. Accordingly, a query-aware LSH function is a random projection coupled with query-aware bucket partition, which removes random shift required by traditional query-oblivious LSH functions. Notably, query-aware bucket partition can be easily implemented so that query performance is guaranteed. We propose a novel query-aware LSH scheme named QALSH for c-ANN search over external memory. Our theoretical studies show that QALSH enjoys a guarantee on query quality. The use of queryaware LSH function enables QALSH to work with any approximation ratio c > 1. Extensive experiments show that QALSH outperforms C2LSH and LSB-Forest, especially in high-dimensional space. Specifically, by using a ratio c < 2, QALSH can achieve much better query quality.

Fast approximate nearest neighbor search with the navigating spreading-out graph

“…Some well-known and widely-used algorithms like the KD-tree [35], R * tree [6], VA-file [38], Locality Sensitive Hashing (LSH) [20], and Product Quantization (PQ) [25] all belong to the above categories. Some works focus on improving the algorithms (e.g., [2,18,19,22,31]), while others focus on optimizing the existing methods according to different platforms and scenarios (e.g., [10,12,36,42]).…”

“…xxx Copyright 2018 VLDB Endowment 2150-8097/18/10... $ 10.00. DOI: https://doi.org/TBD Approximate nearest neighbor search (ANNS) has been a hot topic over decades and provides fundamental support for many applications in data mining, databases, and information retrieval [2,10,12,22,36,42]. For sparse discrete data (like documents), the nearest neighbor search can be carried out efficiently on advanced index structures (e.g., inverted index [34]).…”

“…For sparse discrete data (like documents), the nearest neighbor search can be carried out efficiently on advanced index structures (e.g., inverted index [34]). For dense continuous vectors, various solutions have been proposed such as tree-structure based approaches [2,6,8,17,23,35], hashing-based approaches [18,20,22,31,39], quantization-based approaches [1,19,25,38], and graph-based approaches [3,21,32,40]. Among them, graph-based methods have shown great potential recently.…”

Drawbacks and Proposed Solutions for Real-time Processing on Existing State-of-the-art Locality Sensitive Hashing Techniques

“…Additionally, C2LSH can also determine the number of layers required for a dataset (which depends on the cardinality of the dataset) to satisfy a given success probability. While the memory footprint was much lower and the parameter tuning drawback was solved, the accuracy of C2LSH was still not high [18].…”

“…QALSH [18] was introduced that utilized these two novel techniques to improve upon accuracy (at the slight expense of performance). QALSH introduced query-aware hash functions by creating a B+-tree on each random projection and performing incremental range queries until top-k candidates are found.…”