A Comprehensive Performance Analysis of Apache Hadoop and Apache Spark for Large Scale Data Sets Using HiBench

Abstract

In recent times Big Data analytics has got tremendous attention and it involves storing, processing, and analysing large scale datasets. The advent of distributed computing frameworks such as Hadoop and Spark offers an efficient solution to analyse vast amounts of data. Due to the availability of an application program ming interface (API) and its performance, Spark become very popular, even more popular than the MapReduce framework. Both these frameworks have more than 150 parameters and the combination of these parameters have a huge impact on cluster performance. The system default parameters help the system administrator to deploy their system applications without much effort, and they can measure their specific cluster performance with factory-set parameters. However, an open question remains: can new parameter selection improve cluster performance? In this regard, our study investigates the most impacting parameters such as input splits and shuffling, in order to compare the performance between Hadoop and Spark, using a specific cluster implemented in our department. We used a trial-and-error approach for tuning these parameters based on a large number of experiments. In order to evaluate the frameworks comparison and analysis, we select two work- loads: WordCount and TeraSort. The performance metrics are carried out based on three criteria, namely execution time, throughput, and speedup. Our experimental results revealed that both system performances heavily depends on input data size and correct parameter selection. The analysis results found, unsurprisingly, that Spark has better performance as compared to Hadoop, achieving up to 2 times speedup in WordCount workload and up to 14 times in TeraSort workloads when default parameters are replaced. Finally, we conclude that the system performance depends on different parameters configuration alternatives, and they depend on the data size.

A mini-review of machine learning in big data analytics: Applications, challenges, and prospects

“…It was observed that 58% of the reviewed papers [15, were based on BD predictive analytics, 18% BD prescriptive analytics [103][104][105] , 11% BD descriptive analytics [15, , whiles 9% (A+B) [100][101][102][103][104][105] and 5% (B+C) [106][107][108] . Few studies in BDA used prescriptive analytics (see Table A1 in Appendix); this can be attributed to fact that big data prescriptive analytics is in its early stage.…”

“…Hence, it is not a shock to see such a massive study in the healthcare industry (30%), followed by anomaly detection (11%), cybersecurity, data privacy & IoT (5%) and automobile and transportation (5%); (see Table A1 in Appendix). Concerning the data size, some studies indicated their data size in terms of the storage space ranging from 708 MB [72] to 600 GB [103] , while in terms of the number of observations, it ranges from 1789 [93] to 3 billion records [112] . Based on the data size (e.g., 708 MB [72] ), one can say that the study by Jallad et al [72] is not related to big data.…”

“…This finding can be attributed to Spark being faster and easier to utilize for big data analytics than Hadoop MapReduce. Also, it is believed that Spark offers high processing speed than Hadoop [103] .…”

“…On the other hand, a comparative study shows that Spark consumes more memory in operation than Hadoop since it loads all processes in memory and keeps them in caches for some time [103,114,115] . Therefore, this paper recommends choosing between these two platforms to be grounded on different features.…”