Big data Predictive Analytics for Apache Spark using Machine Learning

Junaid, Muhammad; Wagan, Shiraz Ali; Qureshi, Nawab Muhammad Faseeh; Nam, Choon Sung; Shin, Dong Ryeol

doi:10.1109/gcwot49901.2020.9391620

Cited by 2 publications

(1 citation statement)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MatLab is a numerical computations programming language that is mostly used by engineers and data analysts.It is a unified, strong platform for dealing with Big data [55,56].We may choose from a number of toolboxes to expand on the fundamental capabilities that come with it. Matlab is available for use on Unix, Macintosh, and Windows systems, as well as personal computers for students.…”

Section: Matlabmentioning

confidence: 99%

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Junaid

Ali

Siddiqui

et al. 2022

Wireless Pers Commun

Self Cite

View full text Add to dashboard Cite

Artificial intelligence, specifically machine learning, has been applied in a variety of methods by the research group to transform several data sources into valuable facts and understanding, allowing for superior pattern identification skills. Machine learning algorithms on huge and complicated data sets, computationally expensive on the other hand, processing requires hardware and logical resources, such as space, CPU, and memory. As the amount of data created daily reaches quintillion bytes, A complex big data infrastructure becomes more and more relevant. Apache Spark Machine learning library (ML-lib) is a famous platform used for big data analysis, it includes several useful features for machine learning applications, involving regression, classification, and dimension reduction, as well as clustering and features extraction. In this contribution, we consider Apache Spark ML-lib as a computationally independent machine learning library, which is open-source, distributed, scalable, and platform. We have evaluated and compared several ML algorithms to analyze the platform’s qualities, compared Apache Spark ML-lib against Rapid Miner and Sklearn, which are two additional Big data and machine learning processing platforms. Logistic Classifier (LC), Decision Tree Classifier (DTc), Random Forest Classifier (RFC), and Gradient Boosted Tree Classifier (GBTC) are four machine learning algorithms that are compared across platforms. In addition, we have tested general regression methods such as Linear Regressor (LR), Decision Tree Regressor (DTR), Random Forest Regressor (RFR), and Gradient Boosted Tree Regressor (GBTR) on SUSY and Higgs datasets. Moreover, We have evaluated the unsupervised learning methods like K-means and Gaussian Mixer Models on the data set SUSY and Hepmass to determine the robustness of PySpark, in comparison with the classification and regression models. We used ”SUSY,” ”HIGGS,” ”BANK,” and ”HEPMASS” dataset from the UCI data repository. We also talk about recent developments in the research into Big Data machines and provide future research directions.

show abstract

Section: Matlabmentioning

confidence: 99%

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Junaid

Ali

Siddiqui

et al. 2022

Wireless Pers Commun

Self Cite

View full text Add to dashboard Cite

show abstract

Elevating Smart Manufacturing with a Unified Predictive Maintenance Platform: The Synergy between Data Warehousing, Apache Spark, and Machine Learning

Su,

Huang,

2024

Sensors

View full text Add to dashboard Cite

The transition to smart manufacturing introduces heightened complexity in regard to the machinery and equipment used within modern collaborative manufacturing landscapes, presenting significant risks associated with equipment failures. The core ambition of smart manufacturing is to elevate automation through the integration of state-of-the-art technologies, including artificial intelligence (AI), the Internet of Things (IoT), machine-to-machine (M2M) communication, cloud technology, and expansive big data analytics. This technological evolution underscores the necessity for advanced predictive maintenance strategies that proactively detect equipment anomalies before they escalate into costly downtime. Addressing this need, our research presents an end-to-end platform that merges the organizational capabilities of data warehousing with the computational efficiency of Apache Spark. This system adeptly manages voluminous time-series sensor data, leverages big data analytics for the seamless creation of machine learning models, and utilizes an Apache Spark-powered engine for the instantaneous processing of streaming data for fault detection. This comprehensive platform exemplifies a significant leap forward in smart manufacturing, offering a proactive maintenance model that enhances operational reliability and sustainability in the digital manufacturing era.

show abstract

Big data Predictive Analytics for Apache Spark using Machine Learning

Cited by 2 publications

References 39 publications

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Elevating Smart Manufacturing with a Unified Predictive Maintenance Platform: The Synergy between Data Warehousing, Apache Spark, and Machine Learning

Contact Info

Product

Resources

About