An Enhanced Parallelisation Model for Performance Prediction of Apache Spark on a Multinode Hadoop Cluster

Abstract

Big data frameworks play a vital role in storing, processing, and analysing large datasets. Apache Spark has been established as one of the most popular big data engines for its efficiency and reliability. However, one of the significant problems of the Spark system is performance prediction. Spark has more than 150 configurable parameters, and configuration of so many parameters is challenging task when determining the suitable parameters for the system. In this paper, we proposed two distinct parallelisation models for performance prediction. Our insight is that each node in a Hadoop cluster can communicate with identical nodes, and a certain function of the non-parallelisable runtime can be estimated accordingly. Both models use simple equations that allows us to predict the runtime when the size of the job and the number of executables are known. The proposed models were evaluated based on five HiBench workloads, Kmeans, PageRank, Graph (NWeight), SVM, and WordCount. The workload’s empirical data were fitted with one of the two models meeting the accuracy requirements. Finally, the experimental findings show that the model can be a handy and helpful tool for scheduling and planning system deployment.

Application of Natural Language Processing and Genetic Algorithm to Fine-Tune Hyperparameters of Classifiers for Economic Activities Analysis

“…This diagram shows the prevalence of economic activities across various economic sectors, shedding light on the dynamics of procurement activities within each division. For the efficient processing and analysis of text descriptors at scale, we employed Apache Spark, a high-performance framework for distributed data processing [16][17][18][19]. The approach involved vectorizing descriptor texts using the Term Frequency-Inverse Document Frequency (TF-IDF) method, which assesses the importance of each word in the context of the entire text corpus.…”

A Novel Multi-Task Performance Prediction Model for Spark

“…Matteussi et al [17] provided a comprehensive performance evaluation of Spark Streaming backpressure. Ahmed et al [18] proposed two different parallelization models for performance prediction. Zhu et al in [19] proposed a model to capture the execution behavior of tasks, phases, and jobs, and based on the model, implemented a prototype system.…”

“…Upon computing the dot-product attention scores, the third step involves leveraging these scores to compute the weighted sum of features for each data point, yielding the novel feature representation for each head. For head l (where l ∈ {1, 2, 3, 4}) and data point i, the new feature representation Q li can be acquired using Equation (18).…”