Understanding Vertical Scalability of I/O Virtualization for MapReduce Workloads: Challenges and Opportunities

2014 5th International Workshop on Data-Intensive Computing in the Clouds

Costan

et al. 2014

Self Cite

International audienceResearch on cloud-based Big Data analytics has focused so far on optimizing the performance and cost-effectiveness of the computations, while largely neglecting an important as-pect: users need to upload massive datasets on clouds for their computations. This paper studies the problem of run-ning MapReduce applications when considering the simulta-neous optimization of performance and cost of both the data upload and its corresponding computation taken together. We analyze the feasibility of incremental MapReduce ap-proaches to advance the computation as much as possible during the data upload by using already transferred data to calculate intermediate results. Our key finding shows that overlapping the transfer time with as many incremental com-putations as possible is not always efficient: a better solution is to wait for enough to fill the computational capacity of the MapReduce cluster. Results show significant performance and cost reduction compared with state-of-the-art solutions that leverage incremental computations in a naive fashion

Section: Discussionmentioning

confidence: 99%

To Overlap or Not to Overlap: Optimizing Incremental MapReduce Computations for On-Demand Data Upload

Ene

2014 5th International Workshop on Data-Intensive Computing in the Clouds

Costan

et al. 2014

Self Cite

“…Vertical scalability issues are explored in [8]. With respect to I/O, Ren et al [9] conclude that improving data locality has little potential to improve I/O performance.…”

Section: Related Workmentioning

confidence: 99%

Leveraging Adaptive I/O to Optimize Collective Data Shuffling Patterns for Big Data Analytics

IEEE Trans. Parallel Distrib. Syst.

Costa

Misale³

et al. 2017

Self Cite

Abstract-Big data analytics is an indispensable tool in transforming science, engineering, medicine, health-care, finance and ultimately business itself. With the explosion of data sizes and need for shorter time-to-solution, in-memory platforms such as Apache Spark gain increasing popularity. In this context, data shuffling, a particularly difficult transformation pattern, introduces important challenges. Specifically, data shuffling is a key component of complex computations that has a major impact on the overall performance and scalability. Thus, speeding up data shuffling is a critical goal. To this end, state-of-the-art solutions often rely on overlapping the data transfers with the shuffling phase. However, they employ simple mechanisms to decide how much data and where to fetch it from, which leads to sub-optimal performance and excessive auxiliary memory utilization for the purpose of prefetching. The latter aspect is a growing concern, given evidence that memory per computation unit is continuously decreasing while interconnect bandwidth is increasing. This paper contributes a novel shuffle data transfer strategy that addresses the two aforementioned dimensions by dynamically adapting the prefetching to the computation. We implemented this novel strategy in Spark, a popular in-memory data analytics framework. To demonstrate the benefits of our proposal, we run extensive experiments on an HPC cluster with large core count per node. Compared with the default Spark shuffle strategy, our proposal shows: up to 40% better performance with 50% less memory utilization for buffering and excellent weak scalability.

“…How to optimize the performance of big data applications has been extensively studied in the context of MapReduce. Vertical scalability issued are explored in [6]. Overlapping the map phase with the reduce phase efficiently such that reducers do not lock out resources when idle is explored in [7].…”

Section: Related Workmentioning

confidence: 99%

Towards Memory-Optimized Data Shuffling Patterns for Big Data Analytics

2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)

Costa

Misale

et al. 2016

Self Cite

Big data analytics is an indispensable tool in transforming science, engineering, medicine, healthcare, finance and ultimately business itself. With the explosion of data sizes and need for shorter time-to-solution, in-memory platforms such as Apache Spark gain increasing popularity. However, this introduces important challenges, among which data shuffling is particularly difficult: on one hand it is a key part of the computation that has a major impact on the overall performance and scalability so its efficiency is paramount, while on the other hand it needs to operate with scarce memory in order to leave as much memory available for data caching. In this context, efficient scheduling of data transfers such that it addresses both dimensions of the problem simultaneously is non-trivial. State-of-the-art solutions often rely on simple approaches that yield sub-optimal performance and resource usage. This paper contributes a novel shuffle data transfer strategy that dynamically adapts to the computation with minimal memory utilization, which we briefly underline as a series of design principles.