From Big Data to Smart Data with the K-Nearest Neighbours Algorithm

Abstract

Abstract-The k-nearest neighbours algorithm is one of the most widely used data mining models because of its simplicity and accurate results. However, when it comes to deal with big datasets, with potentially noisy and missing information, this technique becomes ineffective and inefficient. Due to its drawbacks to tackle large amounts of imperfect data, plenty of research has aimed at improving this algorithm by means of data preprocessing techniques. These weaknesses have turned out as strengths and the k-nearest neighbours rule has become a core model to actually detect and correct imperfect data, eliminating noisy and redundant data, as well as correcting missing values.In this work, we delve into the role of the k nearest neighbour algorithm to come up with smart data from big datasets. We analyse how this model is affected by the big data problem, but at the same time, how it can be used to transform raw data into useful data. Concretely, we discuss the benefits of recent big data technologies (Hadoop and Spark) to enable this model to address large amounts of data, as well as the usefulness of prototype reduction and missing values imputation techniques based on it. As a result, guidelines on the use of the k-nearest neighbour to obtain Smart data are provided and new potential research trends are drawn.

Transforming big data into smart data: An insight on the use of the k‐nearest neighbors algorithm to obtain quality data

“…Although most of these data preprocessing techniques were motivated by k‐NN drawbacks, it turns out that the resulting “smart” dataset provided by the above approaches can also be of use in many other learning algorithms Cano, Herrera, and Lozano (); Luengo et al (). This work reviews the current specialized literature that revolves around the idea of the k‐NN to come up with Smart Data, greatly extending our preliminary contribution in Triguero, Maillo, Luengo, García, and Herrera () around this topic. First, we will deepen into the concepts of big and Smart Data and how to extract value from Big Data with existing technologies and Big Data preprocessing techniques (Section 2).…”

A Survey of Data Mining Techniques for Smart Museum Applications

“…This method is widely applied for data classification and pattern recognition [26]. KNN is a high cost technique for have to find all the distances between data [27] and K the centroid to find out the closest distance that represents the similarity of the data to a certain group.…”

Sodobne informacijske tehnologije in storitve (Zbornik dvaindvajsete konference)

“…Omenjeni proces predstavlja eno izmed pomembnih faz podatkovnega rudarjenja, namenjeno čiščenju in korigiranju izvornih podatkov z namenom učinkovitejšega apliciranja algoritmov strojnega učenja (npr. evolucijskih algoritmov, globokega učenja, regresijske analize) [23]. Med opravila predpriprave podatkov štejemo čiščenje (angl.…”