A MapReduce-Based k-Nearest Neighbor Approach for Big Data Classification

Maillo, Jesús; Triguero, Isaac; Herrera, Francisco

doi:10.1109/trustcom.2015.577

Cited by 76 publications

(54 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This approach iteratively performs MapReduce for every single test instance, with the consequent time consumption of Hadoop-based systems for iterations. In [28], however, we proposed a single Hadoop MapReduce process that can simultaneously clas-A C C E P T E D M A N U S C R I P T sify large amounts of test samples against a big training dataset, avoiding start-up costs of Hadoop. To do so, we read the test set line by line from the Hadoop File System, which make this model fully scalable but its performance can be further improved by in-memory solutions.…”

Section: A C C E P T E D Mmentioning

confidence: 99%

“…• We extend the MapReduce scheme proposed in [28] by using multiples reducers to speed up the processing when the number of maps needed is very high.…”

Section: A C C E P T E D Mmentioning

confidence: 99%

“…The latter was proposed in [28], denoted as MR-kNN, in which a single MapReduce process manages the classification of the (big) test set. To do that, Hadoop primitives are used to read line by line the test data within the map phase.…”

Section: Accepted Manuscriptmentioning

confidence: 99%

“…• As in [28] and [27], a MapReduce process will split the training dataset, as it is usually the biggest dataset, into m tasks. In contradistinction to kNN-join approaches that need m 2 tasks, we reduce the complexity of kNN to m tasks without requiring any preprocessing in advanced.…”

Section: Knn-is: An Iterative Spark-based Design Of the Knn Classifiementioning

confidence: 99%

See 3 more Smart Citations

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

Maillo

Ramírez

Triguero

et al. 2017

Knowledge-Based Systems

Self Cite

265

126

View full text Add to dashboard Cite

A note on versions:The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the repository url above for details on accessing the published version and note that access may require a subscription. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.ACCEPTED MANUSCRIPT A C C E P T E D M A N U S C R I P T AbstractThe k-Nearest Neighbors classifier is a simple yet effective widely renowned method in data mining. The actual application of this model in the big data domain is not feasible due to time and memory restrictions. Several distributed alternatives based on MapReduce have been proposed to enable this method to handle large-scale data. However, their performance can be further improved with new designs that fit with newly arising technologies.In this work we provide a new solution to perform an exact k-nearest neighbor classification based on Spark. We take advantage of its in-memory operations to classify big amounts of unseen cases against a big training dataset. The map phase computes the k-nearest neighbors in different training data splits. Afterwards, multiple reducers process the definitive neighbors from the list obtained in the map phase. The key point of this proposal lies on the management of the test set, keeping it in memory when possible. Otherwise, it is split into a minimum number of pieces, applying a MapReduce per chunk, using the caching skills of Spark to reuse the previously partitioned * Corresponding author. ACCEPTED MANUSCRIPT A C C E P T E D M A N U S C R I P T training set. In our experiments we study the differences between Hadoop and Spark implementations with datasets up to 11 million instances, showing the scaling-up capabilities of the proposed approach. As a result of this work an open-source Spark package is available.

show abstract

Section: A C C E P T E D Mmentioning

confidence: 99%

“…• We extend the MapReduce scheme proposed in [28] by using multiples reducers to speed up the processing when the number of maps needed is very high.…”

Section: A C C E P T E D Mmentioning

confidence: 99%

Section: Accepted Manuscriptmentioning

confidence: 99%

Section: Knn-is: An Iterative Spark-based Design Of the Knn Classifiementioning

confidence: 99%

See 2 more Smart Citations

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

Maillo

Ramírez

Triguero

et al. 2017

Knowledge-Based Systems

Self Cite

265

126

View full text Add to dashboard Cite

show abstract

“…In [19] authors proposed an approach of k-NN using Hadoop. First, it splits the T R set and it is distributed over the computing nodes.…”

Section: B K-nn Design For Hadoop and Sparkmentioning

confidence: 99%

From Big Data to Smart Data with the K-Nearest Neighbours Algorithm

Triguero

Maillo

Luengo

et al. 2016

2016 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and I

Self Cite

View full text Add to dashboard Cite

Abstract-The k-nearest neighbours algorithm is one of the most widely used data mining models because of its simplicity and accurate results. However, when it comes to deal with big datasets, with potentially noisy and missing information, this technique becomes ineffective and inefficient. Due to its drawbacks to tackle large amounts of imperfect data, plenty of research has aimed at improving this algorithm by means of data preprocessing techniques. These weaknesses have turned out as strengths and the k-nearest neighbours rule has become a core model to actually detect and correct imperfect data, eliminating noisy and redundant data, as well as correcting missing values.In this work, we delve into the role of the k nearest neighbour algorithm to come up with smart data from big datasets. We analyse how this model is affected by the big data problem, but at the same time, how it can be used to transform raw data into useful data. Concretely, we discuss the benefits of recent big data technologies (Hadoop and Spark) to enable this model to address large amounts of data, as well as the usefulness of prototype reduction and missing values imputation techniques based on it. As a result, guidelines on the use of the k-nearest neighbour to obtain Smart data are provided and new potential research trends are drawn.

show abstract

Enabling data security in data using vertical split with parallel feature selection using meta heuristic algorithms

Selvi¹,

Valarmathi²

2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Big data is the emerging trend in modern science that deals with datasets larger and more complex that cannot be dealt by the traditional data processing techniques. This seems to be the core of current technology and business. In practice, many criteria should be considered in the implementation of this technique. The way of the search space for finding potential subsets of features and prediction performance of classifiers are major important issue. To solve this issue, feature selection methods are introduced in the recent work. In the feature selection algorithm, Non‐deterministic Polynomial (NP) Hard, and searching the space has been becomes more difficult task. To solve this problem, this work provides a new approach toward feature selection based on Vertical Split Group FireFly (VSGFF) algorithm. FF algorithm gets its inspiration from social aspects of real fireflies. At the same time, VSGFF is proposed with the principle of multiple clusters to avoid privacy problem. Finally, Naïve Bayes (NB), K Nearest Neighbor (KNN), and Multi‐Layer Perceptron Neural Network (MLPNN) classification algorithms are proposed for big data classification. Experimental outcomes depicts that proposed technique improves classification accuracy by 4% compared to traditional vertical split firefly algorithm.

show abstract

A MapReduce-Based k-Nearest Neighbor Approach for Big Data Classification

Cited by 76 publications

References 10 publications

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

From Big Data to Smart Data with the K-Nearest Neighbours Algorithm

Enabling data security in data using vertical split with parallel feature selection using meta heuristic algorithms

Contact Info

Product

Resources

About