Cross-Phase Optimization in MapReduce

Heintz, Benjamin; Chandra, Abhishek; Weissman, Jon

doi:10.1007/978-1-4939-1905-5_12

Cited by 10 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hierarchical MapReduce [14] is concerned with compute-intensive MapReduce applications and how to apply multiple distributed clusters to them, but uses clusters and not edge resources. Our recent work [9] is focused more on cross-phase MapReduce optimization, albeit in a wide-area setting. Estimating network paths and forecasting future network conditions are addressed by projects such as NWS [20].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Nebula: Distributed Edge Cloud for Data Intensive Computing

Ryden

Oh²,

Chandra³

et al. 2014

2014 IEEE International Conference on Cloud Engineering

Self Cite

View full text Add to dashboard Cite

Centralized cloud infrastructures have become the de-facto platform for data-intensive computing today. However, they suffer from inefficient data mobility due to the centralization of cloud resources, and hence, are highly unsuited for disperseddata-intensive applications, where the data may be spread at multiple geographical locations. In this paper, we present Nebula: a dispersed cloud infrastructure that uses voluntary edge resources for both computation and data storage. We describe the lightweight Nebula architecture that enables distributed dataintensive computing through a number of optimizations including location-aware data and computation placement, replication, and recovery. We evaluate Nebula's performance on an emulated volunteer platform that spans over 50 PlanetLab nodes distributed across Europe, and show how a common data-intensive computing framework, MapReduce, can be easily deployed and run on Nebula. We show Nebula MapReduce is robust to a wide array of failures and substantially outperforms other wide-area versions based on a BOINC like model.

show abstract

Section: Related Workmentioning

confidence: 99%

“…There are a number of relevant distributed MapReduce projects in the literature [12], [14], [9]. Moon [12] is focused on voluntary resources but not in a wide-area setting.…”

Section: Related Workmentioning

confidence: 99%

Nebula: Distributed Edge Cloud for Data Intensive Computing

Ryden

Oh²,

Chandra³

et al. 2014

2014 IEEE International Conference on Cloud Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [14] the authors focus on the problem of how map and reduce processes are split over hybrid cloud platforms. Heintz et al [15] shows strategies for scheduling map tasks over distributed cloud platforms in order to minimize the performance degradation that can result from large communication times between the platforms. Our own previous work introduces a proposal for data migration (asynchronous rebalancing of the MapReduce storage layer) and scheduling strategies (enforced locality to avoid data pulls over the weak link) that are specifically designed to address the weak link bottleneck in hybrid cloud bursting [2], [16].…”

Section: Related Workmentioning

confidence: 99%

“…TIME FOR THE ORIGINAL (3) AND EXTENDED(15) ON-PREMISE ONLY BASELINE SCENARIOS No HDFS Off-Premise: the MapReduce runtime makes use of the off-premise VMs but HDFS is deployed only on-premise; (2) Blocking Rebalance: HDFS is deployed on the off-premise VMs, the data blocks are rebalanced and then, after this process finished, the MapReduce application is executed; (3) Plain Asynchronous Rebalance: HDFS is deployed on the off-premise VMs and the rebalancing starts in the background at the same time as the application with lo-…”

mentioning

confidence: 99%

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Clemente-Castello

Nicolae

Rafique

et al. 2017

2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

View full text Add to dashboard Cite

Hybrid cloud bursting (i.e., leasing temporary offpremise cloud resources to boost the overall capacity during peak utilization) is a popular and cost-effective way to deal with the increasing complexity of big data analytics. It is particularly promising for iterative MapReduce applications that reuse massive amounts of input data at each iteration, which compensates for the high overhead and cost of concurrent data transfers from the on-premise to the off-premise VMs over a weak inter-site link that is of limited capacity. In this paper we study how to combine various MapReduce data locality techniques designed for hybrid cloud bursting in order to achieve scalability for iterative MapReduce applications in a cost-effective fashion. This is a non-trivial problem due to the complex interaction between the data movements over the weak link and the scheduling of computational tasks that have to adapt to the shifting data distribution. We show that using the right combination of techniques, iterative MapReduce applications can scale well in a hybrid cloud bursting scenario and come even close to the scalability observed in single sites.

show abstract

“…This has led to the emergence of a number of distributed computing frameworks such as Hadoop [5], Dryad [6], Pregel [7], and others [8], [9]. Although these computing frameworks were originally designed for processing applications in a cluster environment, researchers have also looked at adapting and optimizing them in a geo-distributed environment [10]- [12]. We believe that the growth of geo-distributed data and applications that operate on widely distributed data will trigger more computing frameworks to be developed or adapted for a geo-distributed system.…”

Section: Introductionmentioning

confidence: 99%

Awan: Locality-Aware Resource Manager for Geo-Distributed Data-Intensive Applications

Jonathan

Chandra

Weissman

2016

2016 IEEE International Conference on Cloud Engineering (IC2E)

Self Cite

View full text Add to dashboard Cite

Today, many organizations need to operate on data that is distributed around the globe. This is inevitable due to the nature of data that is generated in different locations such as video feeds from distributed cameras, log files from distributed servers, and many others. Although centralized cloud platforms have been widely used for data-intensive applications, such systems are not suitable for processing geo-distributed data due to high data transfer overheads. An alternative approach is to use an Edge Cloud which reduces the network cost of transferring data by distributing its computations globally. While the Edge Cloud is attractive for geo-distributed data-intensive applications, extending existing cluster computing frameworks to a wide-area environment must account for locality. We propose Awan: a new locality-aware resource manager for geo-distributed dataintensive applications. Awan allows resource sharing between multiple computing frameworks while enabling high locality scheduling within each framework. Our experiments with the Nebula Edge Cloud on PlanetLab show that Awan achieves up to a 28% increase in locality scheduling which reduces the average job turnaround time by approximately 20% compared to existing cluster management mechanisms.

show abstract

Cross-Phase Optimization in MapReduce

Cited by 10 publications

References 20 publications

Nebula: Distributed Edge Cloud for Data Intensive Computing

Nebula: Distributed Edge Cloud for Data Intensive Computing

Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce

Awan: Locality-Aware Resource Manager for Geo-Distributed Data-Intensive Applications

Contact Info

Product

Resources

About