Spectral Approaches to Nearest Neighbor Search

Abstract

We study spectral algorithms for the high-dimensional Nearest Neighbor Search problem (NNS). In particular, we consider a semi-random setting where a dataset P in R d is chosen arbitrarily from an unknown subspace of low dimension k d, and then perturbed by fully d-dimensional Gaussian noise. We design spectral NNS algorithms whose query time depends polynomially on d and log n (where n = |P |) for large ranges of k, d and n. Our algorithms use a repeated computation of the top PCA vector/subspace, and are effective even when the random-noise magnitude is much larger than the interpoint distances in P . Our motivation is that in practice, a number of spectral NNS algorithms outperform the random-projection methods that seem otherwise theoretically optimal on worst case datasets. In this paper we aim to provide theoretical justification for this disparity.

Optimal Data-Dependent Hashing for Approximate Near Neighbors

“…We stress that this approach gives improvement for worstcase datasets, which is somewhat unexpected. To put this into a perspective: if one were to assume that the dataset has some special structure, it would be more natural to expect speed-ups with data-dependent hashing: such hashing may adapt to the special structure, perhaps implicitly, as was done in, say, [9,29,1]. However, in our setting there is no assumed structure to adapt to, and hence it is unclear why data-dependent hashing shall help.…”

“…First, we show how to achieve success probability n −ρ , query time n oc(1) , and space and preprocessing time n 1+oc(1) , where ρ = 1 2c 2 −1 + oc (1). Finally, to obtain the final result, one then builds O n ρ copies of the above data structure to amplify the probability of success to 0.99 (as explained in Remark 2).…”

“…The most interesting part of the analysis is lower bounding the probability of success: we need to show that it is at least n −ρ−oc (1) , where ρ = 1 2c 2 −1 . The challenge is that we need to analyze a (somewhat adaptive) random process.…”

“…Suppose that there is a (r, cr, p1, p2)-sensitive partition R of R d , where (p1, p2) ∈ (0, 1) and let ρ = ln(1/p1)/ ln(1/p2). Assume that p1, p2 ≥ 1/n oc(1) , one can sample a partition from R in time n oc(1) , store it in space n oc(1) and perform point location in time n oc (1) . Then there exists a data structure for (c, r)-ANN over a set P ⊆ R d with |P | = n with preprocessing time O(dn 1+oc(1) ), probability of success at least n −ρ−oc(1) , space consumption (in addition to the data points) O(n 1+oc(1) ) and expected query time O(dn oc(1) ).…”

LSH Forest: Practical Algorithms Made Theoretical

“…Most such examples include the algorithms that assume some additional structure in the dataset: such as some notion of low intrinsic dimension [KR02, CNBM01, KL04, BKL06, IN07, Cla06, DF08], or low dimensional data-set with high-dimensional noise [AAKK14]. Most relevant to us is the work of [DS13], which, while mostly focusing on the low intrinsic dimensional datasets, give a generic bound for the worst-case datasets as well.…”