On exploiting data locality for iterative mapreduce applications in hybrid clouds

Clemente-Castello, Francisco J.; Nicolae, Bogdan; Mayo, Rafael; Fernández, Juan Carlos Fernández; Rafique, M. Mustafa

doi:10.1145/3006299.3006329

Cited by 7 publications

(9 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Heintz et al [15] shows strategies for scheduling map tasks over distributed cloud platforms in order to minimize the performance degradation that can result from large communication times between the platforms. Our own previous work introduces a proposal for data migration (asynchronous rebalancing of the MapReduce storage layer) and scheduling strategies (enforced locality to avoid data pulls over the weak link) that are specifically designed to address the weak link bottleneck in hybrid cloud bursting [2], [16]. This work focuses on the evaluation of such techniques and the resulting trade-offs that can be used to establish best practices.…”

Section: Related Workmentioning

confidence: 99%

“…AND STRATEGIES This section briefly introduces the challenges of running iterative MapReduce applications in hybrid cloud bursting scenarios and revisits two complementary strategies discussed in greater detail in our previous work [2].…”

Section: Cloud Bursting Data Locality Challengesmentioning

confidence: 99%

“…The current understanding of the challenges in this particular context and how to address them is limited. Our previous work [2] introduces an analysis of such challenges and potential solutions to address them. We focused our study on MapReduce [3] as a reference analytics framework and pointed out two major issues: the need to replicate the initial input data off-premise to avoid repeated concurrent data transfers over the weak link at each iteration, while adapting the scheduling of computation tasks such that it can take advantage of this dynamic data locality created by the data transfers.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Clemente-Castello

Nicolae

Rafique

et al. 2017

2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

Self Cite

View full text Add to dashboard Cite

Hybrid cloud bursting (i.e., leasing temporary offpremise cloud resources to boost the overall capacity during peak utilization) is a popular and cost-effective way to deal with the increasing complexity of big data analytics. It is particularly promising for iterative MapReduce applications that reuse massive amounts of input data at each iteration, which compensates for the high overhead and cost of concurrent data transfers from the on-premise to the off-premise VMs over a weak inter-site link that is of limited capacity. In this paper we study how to combine various MapReduce data locality techniques designed for hybrid cloud bursting in order to achieve scalability for iterative MapReduce applications in a cost-effective fashion. This is a non-trivial problem due to the complex interaction between the data movements over the weak link and the scheduling of computational tasks that have to adapt to the shifting data distribution. We show that using the right combination of techniques, iterative MapReduce applications can scale well in a hybrid cloud bursting scenario and come even close to the scalability observed in single sites.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Cloud Bursting Data Locality Challengesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Clemente-Castello

Nicolae

Rafique

et al. 2017

2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In our previous work [3], [4], we showed that iterative applications are particularly well positioned to take advantage of hybrid clouds. This class of applications typically reuses a large amount of invariant input data.…”

Section: Introductionmentioning

confidence: 98%

Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Clemente-Castello

Nicolae

Mayo

et al. 2018

IEEE Trans. Parallel Distrib. Syst.

Self Cite

View full text Add to dashboard Cite

Hybrid cloud bursting (i.e., leasing temporary off-premise cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low throughput, high latency network link between the on-premise and off-premise resources ("weak link") makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on hybrid clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a hybrid cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. Not only are the produced estimations accurate in absolute terms compared with experimental results, but they are also up to an order of magnitude more accurate than applying state-of-art estimation approaches originally designed for single-site MapReduce deployments.

show abstract

“…Proceedings [31] Clemente-Castelló, F.J., Nicolae, B., Mustafa, M.R., Mayo, R., Fernández, J.C. On exploiting data locality for iterative mapreduce applications in hybrid clouds. In BDCAT '16: 3rd IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (2016), pp.…”

Section: Conferencementioning

confidence: 99%

Metodología de predicción de rendimiento de aplicaciones MapReduce iterativas sobre una nube híbrida

Castelló¹,

José²

View full text Add to dashboard Cite

On exploiting data locality for iterative mapreduce applications in hybrid clouds

Cited by 7 publications

References 22 publications

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Metodología de predicción de rendimiento de aplicaciones MapReduce iterativas sobre una nube híbrida

Contact Info

Product

Resources

About