Data and task parallelism in ILP using MapReduce

Srinivasan, Ashwin; Faruquie, Tanveer A.; Joshi, Sachindra

doi:10.1007/s10994-011-5245-8

Cited by 50 publications

(28 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Its application in data mining has been widely spread [12,13,14], to the detriment of other parallelization schemes such as Message Passing Interface [15], because of its fault-tolerant mechanism (recommendable for time-consuming tasks) and its ease of use [16]. Despite its unquestionable breakthrough, researchers have found several limitations in Hadoop Mapreduce to design scalable machine learning tools [17].…”

Section: A C C E P T E D Mmentioning

confidence: 99%

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

Maillo

Ramírez

Triguero

et al. 2017

Knowledge-Based Systems

265

126

View full text Add to dashboard Cite

A note on versions:The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the repository url above for details on accessing the published version and note that access may require a subscription. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.ACCEPTED MANUSCRIPT A C C E P T E D M A N U S C R I P T AbstractThe k-Nearest Neighbors classifier is a simple yet effective widely renowned method in data mining. The actual application of this model in the big data domain is not feasible due to time and memory restrictions. Several distributed alternatives based on MapReduce have been proposed to enable this method to handle large-scale data. However, their performance can be further improved with new designs that fit with newly arising technologies.In this work we provide a new solution to perform an exact k-nearest neighbor classification based on Spark. We take advantage of its in-memory operations to classify big amounts of unseen cases against a big training dataset. The map phase computes the k-nearest neighbors in different training data splits. Afterwards, multiple reducers process the definitive neighbors from the list obtained in the map phase. The key point of this proposal lies on the management of the test set, keeping it in memory when possible. Otherwise, it is split into a minimum number of pieces, applying a MapReduce per chunk, using the caching skills of Spark to reuse the previously partitioned * Corresponding author. ACCEPTED MANUSCRIPT A C C E P T E D M A N U S C R I P T training set. In our experiments we study the differences between Hadoop and Spark implementations with datasets up to 11 million instances, showing the scaling-up capabilities of the proposed approach. As a result of this work an open-source Spark package is available.

show abstract

Section: A C C E P T E D Mmentioning

confidence: 99%

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

Maillo

Ramírez

Triguero

et al. 2017

Knowledge-Based Systems

265

126

View full text Add to dashboard Cite

show abstract

“…There are few other rule induction techniques like Inductive Logic Programming (ILP) in MapReduce [40]. ILP generate hypothesis by combining the background knowledge with positive and negative examples from the data and it tries to make relations among the provided data examples with the background domain knowledge.…”

Section: Inductive Logic Programmingmentioning

confidence: 99%

Rule Based Systems in a Distributed Environment: Survey

Bagavathi¹,

Tzacheva²

2017

World Congress on Electrical Engineering and Computer Systems and Science

View full text Add to dashboard Cite

-Over the past years, the internet has become faster, computer storage has become larger and the data from internet users and sensors is piling up to larger amount and is also spread around the globe. This requires more time and space for a single computer to cope with them. Ecosystems like Hadoop helps to store, process and retrieve back the data efficiently in a distributed fashion. Data mining finds substantial improvements over such distributed frameworks to process a large volume of data in a lesser time. Currently, there are many approaches to do data mining tasks such as classification and clustering in a distributed setup using Hadoop MapReduce, Spark, and other Cloud platforms. Actionable pattern mining is a rule based data mining approach for discovering knowledge from information systems in a form of Action Rules. An emphasis of traditional classification rules from a supervised Machine Learning is to predict class label of a data object. Whereas Action Rules produce actionable knowledge in the form of suggestions on how an object can change from one class value to another more desirable class value. This paper gives a brief survey of previous works on association and classification rule mining algorithms in a distributed environment, as well as action rule mining algorithms, and discusses Action Rule Mining in a distributed environment.

show abstract

“…This scheme is currently taken into consideration in data mining, rather than other parallelization schemes such as MPI (Message Passing Interface) [11], because of its faulttolerant mechanism, which is crucial for time-consuming jobs, and because of its simplicity. In the specialized literature, several recent proposals have focused on the parallelization of machine learning tools based on the MapReduce approach [12,13]. For example, some classification techniques such as [14,15,16] have been implemented within the MapReduce paradigm.…”

Section: Introductionmentioning

confidence: 99%

MRPR: A MapReduce solution for prototype reduction in big data classification

et al. 2015

View full text Add to dashboard Cite

In the era of big data, analyzing and extracting knowledge from large-scale data sets is a very interesting and challenging task. The application of standard data mining tools in such data sets is not straightforward. Hence, a new class of scalable mining method that embraces the huge storage and processing capacity of cloud platforms is required. In this work, we propose a novel distributed partitioning methodology for prototype reduction techniques in nearest neighbor classification. These methods aim at representing original training data sets as a reduced number of instances. Their main purposes are to speed up the classification process and reduce the storage requirements and sensitivity to noise of the nearest neighbor rule. However, the standard prototype reduction methods cannot cope with very large data sets. To overcome this limitation, we develop a MapReduce-based framework to distribute the functioning of these algorithms through a cluster of computing elements, proposing several algorithmic strategies to integrate multiple partial solutions (reduced sets of prototypes) into a single one. The proposed model enables prototype reduction algorithms to be applied over big data classification problems without significant accuracy loss. We test the speeding up capabilities of our model with data sets up to 5.7 millions of Email addresses: triguero@decsai.ugr.es (Isaac Triguero), dperalta@decsai.ugr.es (Daniel Peralta), jaume.bacardit@newcastle.ac.uk (Jaume Bacardit), sglopez@ujaen.es (Salvador García), herrera@decsai.ugr.es (Francisco Herrera) March 3, 2014 instances. The results show that this model is a suitable tool to enhance the performance of the nearest neighbor classifier with big data. Preprint submitted to Neurocomputing

show abstract

Data and task parallelism in ILP using MapReduce

Cited by 50 publications

References 23 publications

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data

Rule Based Systems in a Distributed Environment: Survey

MRPR: A MapReduce solution for prototype reduction in big data classification

Contact Info

Product

Resources

About