Effective Selection of Machine Learning Algorithms for Big Data Analytics Using Apache Spark

Hafez, Manar Mohamed; Shehab, Mohamed; Fakharany, Essam El; Hegazy, Abd El Ftah Abdel Ghfar

doi:10.1007/978-3-319-48308-5_66

Cited by 10 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hadoop, Spark, NO-SQL, Sklearn and Weka libraries, Hive, Cloud, and Rapid Miner technologies are gaining popularity. These technologies are computer software tools for extracting, managing, and analyzing data from a massively complex and large data collection that traditional management tools would never be able to handle [29][30][31][32][33][34]. However, in such a setting, selecting among a variety of technologies may be time consuming and difficult.…”

Section: Big Data Technologiesmentioning

confidence: 99%

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Junaid

Ali

Siddiqui

et al. 2022

Wireless Pers Commun

View full text Add to dashboard Cite

Artificial intelligence, specifically machine learning, has been applied in a variety of methods by the research group to transform several data sources into valuable facts and understanding, allowing for superior pattern identification skills. Machine learning algorithms on huge and complicated data sets, computationally expensive on the other hand, processing requires hardware and logical resources, such as space, CPU, and memory. As the amount of data created daily reaches quintillion bytes, A complex big data infrastructure becomes more and more relevant. Apache Spark Machine learning library (ML-lib) is a famous platform used for big data analysis, it includes several useful features for machine learning applications, involving regression, classification, and dimension reduction, as well as clustering and features extraction. In this contribution, we consider Apache Spark ML-lib as a computationally independent machine learning library, which is open-source, distributed, scalable, and platform. We have evaluated and compared several ML algorithms to analyze the platform’s qualities, compared Apache Spark ML-lib against Rapid Miner and Sklearn, which are two additional Big data and machine learning processing platforms. Logistic Classifier (LC), Decision Tree Classifier (DTc), Random Forest Classifier (RFC), and Gradient Boosted Tree Classifier (GBTC) are four machine learning algorithms that are compared across platforms. In addition, we have tested general regression methods such as Linear Regressor (LR), Decision Tree Regressor (DTR), Random Forest Regressor (RFR), and Gradient Boosted Tree Regressor (GBTR) on SUSY and Higgs datasets. Moreover, We have evaluated the unsupervised learning methods like K-means and Gaussian Mixer Models on the data set SUSY and Hepmass to determine the robustness of PySpark, in comparison with the classification and regression models. We used ”SUSY,” ”HIGGS,” ”BANK,” and ”HEPMASS” dataset from the UCI data repository. We also talk about recent developments in the research into Big Data machines and provide future research directions.

show abstract

Section: Big Data Technologiesmentioning

confidence: 99%

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Junaid

Ali

Siddiqui

et al. 2022

Wireless Pers Commun

View full text Add to dashboard Cite

show abstract

“…It is made to modify the best algorithm. According to a previous study [25], which presents a comparative study among different algorithms with different types and sizes of datasets, the best algorithm is DT; specifically, the classification and regression trees.…”

Section: Proposed Frameworkmentioning

confidence: 99%

Self-Tuning Parameters for Decision Tree Algorithm Based on Big Data Analytics

Hafez¹,

Elfakharany²,

Abohany³

et al. 2023

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Big data is usually unstructured, and many applications require the analysis in real-time. Decision tree (DT) algorithm is widely used to analyze big data. Selecting the optimal depth of DT is time-consuming process as it requires many iterations. In this paper, we have designed a modified version of a (DT). The tree aims to achieve optimal depth by self-tuning running parameters and improving the accuracy. The efficiency of the modified (DT) was verified using two datasets (airport and fire datasets). The airport dataset has 500000 instances and the fire dataset has 600000 instances. A comparison has been made between the modified (DT) and standard (DT) with results showing that the modified performs better. This comparison was conducted on multi-node on Apache Spark tool using Amazon web services. Resulting in accuracy with an increase of 6.85% for the first dataset and 8.85% for the airport dataset. In conclusion, the modified DT showed better accuracy in handling different-sized datasets compared to standard DT algorithm.

show abstract

“…This will be very useful for retailers, who can audit the placement of products, and, for costumers, who may obtain additional information about products by taking a simple picture of the shelf. These approaches usually rely on computer vision techniques [9][10][11][12] and face challenging issues such as the similar appearance in terms of shape, color, texture, and the size of different products. In order to overcome these problems, other research combines the information obtained from image analysis with other information based on statistical methods in order to provide a fine-grained retail product recognition and classification [13].…”

Section: Related Workmentioning

confidence: 99%

Classification of Retail Products: From Probabilistic Ranking to Neural Networks

et al. 2021

Self Cite

View full text Add to dashboard Cite

Food retailing is now on an accelerated path to a success penetration into the digital market by new ways of value creation at all stages of the consumer decision process. One of the most important imperatives in this path is the availability of quality data to feed all the process in digital transformation. However, the quality of data are not so obvious if we consider the variety of products and suppliers in the grocery market. Within this context of digital transformation of grocery industry, Midiadia is a Spanish data provider company that works on converting data from the retailers’ products into knowledge with attributes and insights from the product labels that is maintaining quality data in a dynamic market with a high dispersion of products. Currently, they manually categorize products (groceries) according to the information extracted directly (text processing) from the product labelling and packaging. This paper introduces a solution to automatically categorize the constantly changing product catalogue into a 3-level food taxonomy. Our proposal studies three different approaches: a score-based ranking method, traditional machine learning algorithms, and deep neural networks. Thus, we provide four different classifiers that support a more efficient and less error-prone maintenance of groceries catalogues, the main asset of the company. Finally, we have compared the performance of these three alternatives, concluding that traditional machine learning algorithms perform better, but closely followed by the score-based approach.

show abstract

Effective Selection of Machine Learning Algorithms for Big Data Analytics Using Apache Spark

Cited by 10 publications

References 10 publications

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Self-Tuning Parameters for Decision Tree Algorithm Based on Big Data Analytics

Classification of Retail Products: From Probabilistic Ranking to Neural Networks

Contact Info

Product

Resources

About