Efficient Model Selection for Large-Scale Nearest-Neighbor Data Mining

Abstract

Abstract. One of the most widely used models for large-scale data mining is the k-nearest neighbor (k-nn) algorithm. It can be used for classification, regression, density estimation, and information retrieval. To use k-nn, a practitioner must first choose k, usually selecting the k with the minimal loss estimated by cross-validation. In this work, we begin with an existing but little-studied method that greatly accelerates the cross-validation process for selecting k from a range of user-provided possibilities. The result is that a much larger range of k values may be examined more quickly. Next, we extend this algorithm with an additional optimization to provide improved performance for locally linear regression problems. We also show how this method can be applied to automatically select the range of k values when the user has no a priori knowledge of appropriate bounds. Furthermore, we apply statistical methods to reduce the number of examples examined while still finding a likely best k, greatly improving performance for large data sets. Finally, we present both analytical and experimental results that demonstrate these benefits.

Test Point Specific k Estimation for kNN Classifier

“…We now mention some important results on kNN with variable k. In [21], Hand and Vinciotti showed how k can be chosen only for a two-class classifier where the classes are unbalanced. In [22], Hammerly and Speegle demonstrated an efficient method for k selection based on cross-validation strategies. Song et al in [23] proposed two simple extensions of the kNN method using the concept of informativeness but prior crossvalidation of the datasets for choosing the optimal value of k. Moreover, the authors in [23] do not explicitly take into account the variations in local densities of training points surrounding a test point.…”

Imputing Missing Entries of a Data Matrix: A Review

“…Assessment of model's prediction error, tuning model's parameters, and model selection, e.g., [237] and [244]. In particular, it is often used to determine the number of neighbors in k-Nearest Neighbors models, e.g., [121] and [199]. In the next section we illustrate the use of Hold-out validation to determine a suitable matrix rank when solving least squares problems.…”