Asymmetric Minwise Hashing for Indexing Binary Inner Products and Set Containment

Abstract

Minwise hashing (Minhash) is a widely popular indexing scheme in practice. Minhash is designed for estimating set resemblance and is known to be suboptimal in many applications where the desired measure is set overlap (i.e., inner product between binary vectors) or set containment. Minhash has inherent bias towards smaller sets, which adversely affects its performance in applications where such a penalization is not desirable. In this paper, we propose asymmetric minwise hashing (MH-ALSH), to provide a solution to this well-known problem. The new scheme utilizes asymmetric transformations to cancel the bias of traditional minhash towards smaller sets, making the final "collision probability" monotonic in the inner product. Our theoretical comparisons show that, for the task of retrieving with binary inner products, asymmetric minhash is provably better than traditional minhash and other recently proposed hashing algorithms for general inner products. Thus, we obtain an algorithmic improvement over existing approaches in the literature. Experimental evaluations on four publicly available highdimensional datasets validate our claims. The proposed scheme outperforms, often significantly, other hashing algorithms on the task of near neighbor retrieval with set containment. Our proposal is simple and easy to implement in practice.

BagMinHash - Minwise Hashing Algorithm for Weighted Sets

“…As example, weighted minwise hashing was successfully applied to train generalized min-max kernel support vector machines [18,22]. Furthermore, asymmetric locality-sensitive hashing [33], which can be realized using weighted minwise hashing, was used for e cient deep learning [34]. Finally, it could also be applied to random forests that are constructed using the weighted Jaccard index as similarity measure [30].…”

When Hashes Met Wedges

“…WHIMP gets a precision and recall above 0.7 for at least 75% of the sample. We stress the low values of cosine similarities here: a similarity of 0.2 is well-below the values studied in recent LSH-based results [37,39,38]. It is well-known that low similarity values are harder to detect, yet WHIMP gets accurate results for an overwhelming majority of the vertices/users.…”

GB-KMV: An Augmented KMV Sketch for Approximate Containment Similarity Search

“…Compared with symmetric similarity such as Jaccard similarity, containment similarity gives special consideration on the query size, which makes it more suitable in some applications. As shown in [35], containment similarity is useful in record matching application. Given two text descriptions of two restaurants X and Y which are represented by two "set of words" records: {five, guys, burgers, and, fries, downtown, brooklyn, new, york} and {five, kitchen, berkeley} respectively.…”

“…Challenges. The problem of containment similarity search has been intensively studied in the literature in recent years (e.g., [5], [35], [44]). The key challenges of this problem come from the following three aspects: (i) The number of elements (i.e., vocabulary size) may be very large.…”

“…The key challenges of this problem come from the following three aspects: (i) The number of elements (i.e., vocabulary size) may be very large. For instance, the vocabulary will blow up quickly when the higher-order shingles are used [35]. Moreover, query and record may contain many elements.…”