A Deep Non-linear Feature Mapping for Large-Margin kNN Classification

Abstract

KNN is one of the most popular data mining methods for classification, but it often fails to work well with inappropriate choice of distance metric or due to the presence of numerous class-irrelevant features. Linear feature transformation methods have been widely applied to extract classrelevant information to improve kNN classification, which is very limited in many applications. Kernels have also been used to learn powerful non-linear feature transformations, but these methods fail to scale to large datasets. In this paper, we present a scalable non-linear feature mapping method based on a deep neural network pretrained with Restricted Boltzmann Machines for improving kNN classification in a large-margin framework, which we call DNet-kNN. DNet-kNN can be used for both classification and for supervised dimensionality reduction. The experimental results on two benchmark handwritten digit datasets and one newsgroup text dataset show that DNet-kNN has much better performance than large-margin kNN using a linear mapping and kNN based on a deep autoencoder pretrained with Restricted Boltzmann Machines.

Class Confidence Weighted kNN Algorithms for Imbalanced Data Sets

“….127 (8) .156(3) .123 (10) .156(3) .124 (9) .128(7) .141(5) .140 (6) .619 (2) .699(1) Arrhythmia .160 (7) .214(4) .167 (6) .229(3) .083 (10) .134 (9) .187(5) .156 (8) .346 (2) .385(1) Balance .127(7) .130(5) .145 (2) .149(1) .135 (4) .091 (9) .129 (6) .142(3) .092 (8) .089(10) Cleveland .889 (8) .897 (2) .890 (6) .897 (1) .889 (7) .895(3) .893 (5) .893(4) .806 (10) .846(9) Cmc .346 (9) .383 (2) .357 (7) .384(1) .358 (6) .341 (10) .365 (5) .371(4) .356 (8) .380(3) Credit .888 (7) .895 (2) .887 (8) .894(3) .891 (5) .903(1) .891 (6) .893 (4) .871 (9) .868(10) Ecoli .943(3) .948 (1) .938 (5) .941 (4) .926 (7) .920 (8) .945 (2) .933 (6) .566 (10) .584(9) German .535 (7) .541 (2) .533 …”

“…All experiments are carried out using 5×2 folds cross-validations, and the final results are the average of the repeated runs. (8) .037(4) .036 (6) .043(1) .040(3) .036 (5) .035(7) .042(2) .024 (10) .025(9) Churn .101(7) .113(2) .101 (6) .115(1) .108(4) .100 (8) .107(5) .111(3) .092 (10) .099(9) Upselling .219 (8) .243 (5) .218 (9) .241(6) .288(3) .212 (10) .231(7) .264(4) .443(1) .437(2) Ada.agnostic .641 (9) .654(5) .646 (8) .652 (6) .689(3) .636 (10) .648 (7) .670(4) .723(1) .691(2) Ada.prior .645 (8) .669 (2) .654(7) .668(3) .661 (5) .639 (9) .657 (6) .664(4) .682(1) .605(10) Sylva.agnostic .930 (2) .926 (8) .930(3) .925 (9) .928(6) .922 (10) .928(4) .926 (7) .934(1) .928(5) Sylva.prior .965(4) .965(2) .965 (6) .965(4) .904 (10) .974(1) .965(3) .935 (9) .946 (8) .954(7) BrazilTourism .176 (9) .242(1) .232(5) .241 (2) .233(4) .184 (8) .209 (6) .237(3) .152 (10) .199 (7) Marketing .112 (10) .157 (2) .113 (9) .161(1) .124 (8) .150(3) .134(5) .142(4) .130 (6) .125(7) Backache .311(7) .325(3) .307 (8) .328(2) .317 …”

“…Experimental results of LMNN show large improvements over conventional k NN and SVM. Min et al [8] have proposed DNet which uses a non-linear feature mapping method pre-trained with Restricted Boltzmann Machines to achieve the goal of large-margin k NN classification. Recently, a new method WDk NN was introduced in [9] which discovers optimal weights for each instance in training phase which are taken into account during test phases.…”

Human Sit Down Position Detection Using Data Classification and Dimensionality Reduction

“…It depends on a metric used as a distance to measure how similar are the data points. Here we consider the Euclidean distance [11]. The data matrix N (test set) which is j x l, where j=100 and l=4.…”

“…Such a stage involves a classification step using the well-known K-Nearest Neighbors (K-NN) multilabel classifier that depends of a metric used for the calculus of the Euclidian distance between two points for select the k neighbors nearest in terms of p attributes [11]. As well, due to the high dimension of acquired data (four dimensions) , we perform a DR step based on Principal Component Analysis (PCA) [8].…”