Performance Characteristics of a Cost-Effective Medium-Sized Beowulf Cluster Supercomputer

Barczak, Andre L. C.; Messom, Chris; Johnson, Martin

doi:10.1007/3-540-44864-0_109

Cited by 6 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experimental cluster has its dedicated networking infrastructure, with dedicated switches. The cluster was designed and deployed by a group of experienced academics who previously built Beowulf clusters with optimised performance [32]. This infrastructure is isolated from any other machine to reduce unwanted competition for network resources.…”

Section: Methodsmentioning

confidence: 99%

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

et al. 2021

View full text Add to dashboard Cite

This article proposes a new parallel performance model for different workloads of Spark Big Data applications running on Hadoop clusters. The proposed model can predict the runtime for generic workloads as a function of the number of executors, without necessarily knowing how the algorithms were implemented. For a certain problem size, it is shown that a model based on serial boundaries for a 2D arrangement of executors can fit the empirical data for various workloads. The empirical data was obtained from a real Hadoop cluster, using Spark and HiBench. The workloads used in this work were included WordCount, SVM, Kmeans, PageRank and Graph (Nweight). A particular runtime pattern emerged when adding more executors to run a job. For some workloads, the runtime was longer with more executors added. This phenomenon is predicted with the new model of parallelisation. The resulting equation from the model explains certain performance patterns that do not fit Amdahl’s law predictions, nor Gustafson’s equation. The results show that the proposed model achieved the best fit with all workloads and most of the data sizes, using the R-squared metric for the accuracy of the fitting of empirical data. The proposed model has advantages over machine learning models due to its simplicity, requiring a smaller number of experiments to fit the data. This is very useful to practitioners in the area of Big Data because they can predict runtime of specific applications by analysing the logs. In this work, the model is limited to changes in the number of executors for a fixed problem size.

show abstract

Section: Methodsmentioning

confidence: 99%

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The experimental big data cluster used in this work was designed and developed by a group of academics at Massey University, Auckland campus [38]. The hardware for this experimental big data cluster is similar to a Beowful cluster.…”

Section: Methodsmentioning

confidence: 99%

An Enhanced Parallelisation Model for Performance Prediction of Apache Spark on a Multinode Hadoop Cluster

Ahmed

Barczak

Rashid

et al. 2021

BDCC

View full text Add to dashboard Cite

Big data frameworks play a vital role in storing, processing, and analysing large datasets. Apache Spark has been established as one of the most popular big data engines for its efficiency and reliability. However, one of the significant problems of the Spark system is performance prediction. Spark has more than 150 configurable parameters, and configuration of so many parameters is challenging task when determining the suitable parameters for the system. In this paper, we proposed two distinct parallelisation models for performance prediction. Our insight is that each node in a Hadoop cluster can communicate with identical nodes, and a certain function of the non-parallelisable runtime can be estimated accordingly. Both models use simple equations that allows us to predict the runtime when the size of the job and the number of executables are known. The proposed models were evaluated based on five HiBench workloads, Kmeans, PageRank, Graph (NWeight), SVM, and WordCount. The workload’s empirical data were fitted with one of the two models meeting the accuracy requirements. Finally, the experimental findings show that the model can be a handy and helpful tool for scheduling and planning system deployment.

show abstract

A HyperTransport-based personal parallel computer

Yang

Chen

Cheng

et al. 2008

2008 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

Performance Characteristics of a Cost-Effective Medium-Sized Beowulf Cluster Supercomputer

Cited by 6 publications

References 5 publications

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters

An Enhanced Parallelisation Model for Performance Prediction of Apache Spark on a Multinode Hadoop Cluster

A HyperTransport-based personal parallel computer

Contact Info

Product

Resources

About