Shared Nearest Neighbor clustering in a Locality Sensitive Hashing framework

Abstract

We present a new algorithm to cluster high dimensional sequence data, and its application to the field of metagenomics, which aims to reconstruct individual genomes from a mixture of genomes sampled from an environmental site, without any prior knowledge of reference data (genomes) or the shape of clusters. Such problems typically cannot be solved directly with classical approaches seeking to estimate the density of clusters, e.g., using the shared nearest neighbors rule, due to the prohibitive size of contemporary sequence datasets. We explore here a new method based on combining the shared nearest neighbor (SNN) rule with the concept of Locality Sensitive Hashing (LSH). The proposed method, called LSH-SNN, works by randomly splitting the input data into smaller-sized subsets (buckets) and employing the shared nearest neighbor rule on each of these buckets. Links can be created among neighbors sharing a sufficient number of elements, hence allowing clusters to be grown from linked elements. LSH-SNN can scale up to larger datasets consisting of millions of sequences, while achieving high accuracy across a variety of sample sizes and complexities. * T. Brüls works at Shared Nearest Neighbor clustering in a Locality Sensitive Hashing framework bias is known as the uniform effect of the K-means. Moreover, the number of clusters K has to be specified a priori, which is not trivial when no prior knowledge is available. To address these problems, methods based on estimating the density and/or the similarity among instances have been introduced [15,30].In [14], the authors presented an effective clustering method based on two key notions: the similarity between neighboring elements and the density around instances. This method, Shared Nearest Neighbors (SNN), is a density-based clustering method and incorporates a suitable similarity measure to cluster data. After finding the nearest neighbors of each element and computing the similarity between pairs of points, SNN identifies core points, eliminates noisy elements and builds clusters around the core elements. This method can yield better performance compared to other clustering approaches with data of varying densities, and it can automatically handle the number of output clusters. However, this method has complexity O(n 2 ), where n is the number of instances in the dataset, arising from the computation of the similarity matrix, which can be prohibitive when dealing with high dimensional data.One interesting concept to reduce the burden of computing the similarity matrix is Locality Sensitive Hashing (LSH). This concept was initially introduced to find approximate near neighbor information in high dimensional space [19,51].The key idea is to hash elements into different buckets; then for a query instance x, to use instances stored in buckets containing x as candidates for near neighbors. This approximation reduces the query time complexity to O(log n) instead of O(n) (O(n) is the complexity for searching nearest neighbors for one instance). Therefore, the similarity m...

Enhancement of Short Text Clustering by Iterative Classification

“…Nearest neighbors' based methods for similarity matrix sparsification include k-nearest neighbor [7] and shared nearest neighbor [14]. k-nearest neighbor sparsification keeps only the k highest similarity scores for each text; the sharednearest neighbor approach adds a condition that texts retaining similarity values with a particular text should share a prescribed number of neighbors.…”

Progression of chronic kidney disease in African Americans with type 2 diabetes mellitus using topology learning in electronic medical records

“…That is, for any distinct trajectory pattern, when there are more than 150 patients in the cohort sharing this pattern, the prediction for new patients who also have this trajectory pattern will be accurate. Since patient numbers in EMR data can easily reach into the million range, it is promising to build a large database of trajectory patterns in existing patients and use locality sensitive hashing (28)(29)(30)(31) and approximate nearest neighbor query techniques (32,33) to predict disease progressions in new patients. clusters color-coded by kidney functions (blue: good, red: impaired); clusters enriched with encounters of patients of high-or low-risk APOL1 genotypes (middle and right, correspondingly).…”