A Survey on Geographically Distributed Big-Data Processing Using MapReduce

Dolev, Shlomi; Florissi, Patricia; Gudes, Ehud; Sharma, Shantanu; Singer, Ido

doi:10.1109/tbdata.2017.2723473

Cited by 56 publications

(36 citation statements)

References 119 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The literature is huge, but mostly on distributed system architecture, computing platforms, or data query tools. For a review of recent developments, please see [13,21] and references therein.…”

Section: Related Workmentioning

confidence: 99%

Fast Communication-Efficient Spectral Clustering over Distributed Data

Yan

Wang

et al. 2021

IEEE Trans. Big Data

View full text Add to dashboard Cite

The last decades have seen a surge of interests in distributed computing thanks to advances in clustered computing and big data technology. Existing distributed algorithms typically assume all the data are already in one place, and divide the data and conquer on multiple machines. However, it is increasingly often that the data are located at a number of distributed sites, and one wishes to compute over all the data with low communication overhead. For spectral clustering, we propose a novel framework that enables its computation over such distributed data, with "minimal" communications while a major speedup in computation. The loss in accuracy is negligible compared to the non-distributed setting. Our approach allows local parallel computing at where the data are located, thus turns the distributed nature of the data into a blessing; the speedup is most substantial when the data are evenly distributed across sites. Experiments on synthetic and large UC Irvine datasets show almost no loss in accuracy with our approach while about 2x speedup under various settings with two distributed sites. As the transmitted data need not be in their original form, our framework readily addresses the privacy concern for data sharing in distributed computing.

show abstract

Section: Related Workmentioning

confidence: 99%

Fast Communication-Efficient Spectral Clustering over Distributed Data

Yan

Wang

et al. 2021

IEEE Trans. Big Data

View full text Add to dashboard Cite

show abstract

“…Presently, many big data workloads operate across isolated data stores that are distributed geographically and manipulated by different clouds. For example, the typical scientific data processing pipeline [26,40] consists of multiple stages that are frequently conducted by different research organizations with varied computing demands. Accordingly, accelerating data analysis for each stage may require computing facilities that are located in different clouds.…”

Section: Introductionmentioning

confidence: 99%

“…"Data diffusion" [19,20], which can acquire compute and storage resources dynamically, replicate data in response to demand, and schedule computations close to data, has been proposed for Grid computing. In the cloud era, the idea has been extended to scientific workflows that schedule compute tasks [40] and move required data across the global deployment of cloud centers. Both data diffusion and cloud workflows rely on a centralized site that provides data-aware compute tasks scheduling and supports an index service to locate data sets dispersed globally.…”

Section: Introductionmentioning

confidence: 99%

A Cache-Based Data Movement Infrastructure for On-demand Scientific Cloud Computing

Abramson

Carroll

Jin

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

As cloud computing has become the de facto standard for big data processing, there is interest in using a multi-cloud environment that combines public cloud resources with private on-premise infrastructure. However, by decentralizing the infrastructure, a uniform storage solution is required to provide data movement between different clouds to assist on-demand computing. This paper presents a solution based on our earlier work, the MeDiCI (Metropolitan Data Caching Infrastructure) architecture. Specially, we extend MeDiCI to simplify the movement of data between different clouds and a centralized storage site. It uses a hierarchical caching system and supports most popular infrastructure-as-a-service (IaaS) interfaces, including Amazon AWS and OpenStack. As a result, our system allows the existing parallel data intensive application to be offloaded into IaaS clouds directly. The solution is illustrated using a large bioinformatics application, a Genome Wide Association Study (GWAS), with Amazons AWS, HUAWEI Cloud, and a private centralized storage system. The system is evaluated on Amazon AWS and the Australian national cloud.

show abstract

“…Generally, the data center (DC)-based computing infrastructure serves as an effective platform for satisfying both the Peng Zhao, Xinyu Yang computational and data storage requirements of big data analytics. To meet increasing data analysis demands and provide reliability, service providers deploy their data analytics service globally on multiple geographically distributed DCs, referred to as the Geographically-distributed Data Analytics (GDA) [2], [3]. The basic infrastructures for GDA generally consist of a massive number of servers and multiple Internet Data Centers (IDCs) in different locations.…”

Section: Introductionmentioning

confidence: 99%

Energy-Efficient Analytics for Geographically Distributed Big Data

Zhao

Yang

Lin

et al. 2019

IEEE Internet Comput.

View full text Add to dashboard Cite

Big data analytics on geographically distributed datasets (across data centers or clusters) has been attracting increasing interests in both academia and industry, posing significant complications for system and algorithm design. In this article, we systematically investigate the geo-distributed big-data analytics framework by analyzing the fine-grained paradigm and the key design principles. We present a dynamic global manager selection algorithm (GMSA) to minimize energy consumption cost by fully exploiting the system diversities in geography and variation over time. The algorithm makes real-time decisions based on the measurable system parameters through stochastic optimization methods, while achieving the performance balances between energy cost and latency. Extensive trace-driven simulations verify the effectiveness and efficiency of the proposed algorithm. We also highlight several potential research directions that remain open and require future elaborations in analyzing geo-distributed big data.

show abstract

A Survey on Geographically Distributed Big-Data Processing Using MapReduce

Cited by 56 publications

References 119 publications

Fast Communication-Efficient Spectral Clustering over Distributed Data

Fast Communication-Efficient Spectral Clustering over Distributed Data

A Cache-Based Data Movement Infrastructure for On-demand Scientific Cloud Computing

Energy-Efficient Analytics for Geographically Distributed Big Data

Contact Info

Product

Resources

About