Streaming similarity search over one billion tweets using parallel locality-sensitive hashing

Abstract

Finding nearest neighbors has become an important operation on databases, with applications to text search, multimedia indexing, and many other areas. One popular algorithm for similarity search, especially for high dimensional data (where spatial indexes like kdtrees do not perform well) is Locality Sensitive Hashing (LSH), an approximation algorithm for finding similar objects.In this paper, we describe a new variant of LSH, called Parallel LSH (PLSH) designed to be extremely efficient, capable of scaling out on multiple nodes and multiple cores, and which supports highthroughput streaming of new data. Our approach employs several novel ideas, including: cache-conscious hash table layout, using a 2-level merge algorithm for hash table construction; an efficient algorithm for duplicate elimination during hash-table querying; an insert-optimized hash table structure and efficient data expiration algorithm for streaming data; and a performance model that accurately estimates performance of the algorithm and can be used to optimize parameter settings. We show that on a workload where we perform similarity search on a dataset of > 1 Billion tweets, with hundreds of millions of new tweets per day, we can achieve query times of 1-2.5 ms. We show that this is an order of magnitude faster than existing indexing schemes, such as inverted indexes. To the best of our knowledge, this is the fastest implementation of LSH, with table construction times up to 3.7× faster and query times that are 8.3× faster than a basic implementation.

Drawbacks and Proposed Solutions for Real-time Processing on Existing State-of-the-art Locality Sensitive Hashing Techniques

“…While there are several real-time/streaming applications in high-dimensional spaces that leverage LSH, there have been few works [29,30,31] that aim at improving LSH for streaming applications. In [29], the authors present a parallel LSH framework that is designed to handle similarity searches on incoming twitter data. The goal of their cache-conscious model is to improve on the creation and updation of the hash tables (which are based on the original LSH design).…”

SigCO: Mining significant correlations via a distributed real-time computation engine

“…The second element computes the correlations, performs the significance test and forwards qualified pairs to the last element, where duplicate removal is performed. LSHC: LSHC is based on locality sensitive hashing (LSH) [21], which use the property that the normalized sliding windows of significant correlated time series are close in Euclidean space (refer Section IV). The topology of LSHC consists of three processing elements.…”

“…Sliding windows that are mapped to a bucket in each hash table are shuffled to the same task of the second element, where the correlation computation is performed over the sliding windows in each bucket per hash table. LSHC parameters are chosen to minimize the processing latency while ensuring the failure probability (i.e., the probability of not reporting a certain qualified pair) at 5% [21]. Likewise, the last element aggregates correlated time series pairs…”

“…For the sliding window size h in Figure 4(d), SigCO has nearly 3x and 8x lower cost as compared to GC and DFTC at the highest level of sliding window size. Specifically, because LSHCQ requires a large number of hash tables to achieve low failure rate [21], it incurs high communication cost.…”

Quotient Filters: Approximate Membership Queries on the GPU

“…Since many databases, networks, and file systems benefit from the quick filtering of negative queries (often to avoid costly disk or network accesses), AMQs have found wide use. Such applications are emerging research areas on GPUs [2], [3], [4].…”