Improving Hadoop Service Provisioning in a Geographically Distributed Cloud

Zhang, Qi; Liu, Ling; Lee, Kisung; Zhou, Yang; Singh, Aameek; Mandagere, Nagapramod; Gopisetty, Sandeep; Alatorre, Gabriel

doi:10.1109/cloud.2014.65

Cited by 25 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We build datacenters geographically with the purpose of achieving low latencies for local users [7] . Nevertheless, as data volumes keep increasing at a tremendous rate, it is still time consuming to transfer such substantial amount of data across datacenters [8] . Many cloud services have very stringent requirements for latency, even a delay of one second can make a great difference [9] .…”

Section: Optimization Issuesmentioning

confidence: 99%

Wide area analytics for geographically distributed datacenters

Ji¹,

Li²

2016

Tinshhua Sci. Technol.

View full text Add to dashboard Cite

Big data analytics, the process of organizing and analyzing data to get useful information, is one of the primary uses of cloud services today. Traditionally, collections of data are stored and processed in a single datacenter. As the volume of data grows at a tremendous rate, it is less efficient for only one datacenter to handle such large volumes of data from a performance point of view. Large cloud service providers are deploying datacenters geographically around the world for better performance and availability. A widely used approach for analytics of geo-distributed data is the centralized approach, which aggregates all the raw data from local datacenters to a central datacenter. However, it has been observed that this approach consumes a significant amount of bandwidth, leading to worse performance. A number of mechanisms have been proposed to achieve optimal performance when data analytics are performed over geo-distributed datacenters. In this paper, we present a survey on the representative mechanisms proposed in the literature for wide area analytics. We discuss basic ideas, present proposed architectures and mechanisms, and discuss several examples to illustrate existing work.We point out the limitations of these mechanisms, give comparisons, and conclude with our thoughts on future research directions.

show abstract

Section: Optimization Issuesmentioning

confidence: 99%

Wide area analytics for geographically distributed datacenters

Ji¹,

Li²

2016

Tinshhua Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…As multi-cloud environments become more and more accessible, deploying MapReduce applications over multi-cloud is emerging [5], [6], [7], [8].…”

Section: Introductionmentioning

confidence: 99%

“…First, existing work on multi-cloud mainly focuses on maximizing throughput by improving data locality [5], [8], but the perspective of cost optimization is missing. To consider the cost and running time of applications doing the optimization process, we need to focus not only on data locality, but also on data transfer costs defined by multiple different cloud providers' cost models.…”

Section: Introductionmentioning

confidence: 99%

Cost-Efficient High-Performance Internet-Scale Data Analytics over Multi-cloud Environments

Imai

Patterson

Varela

2015

2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

Abstract-To analyze data distributed across the world, one can use distributed computing power to take advantage of data locality and achieve higher throughput. The multi-cloud model, a composition of multiple clouds, can provide cost-effective computing resources to process such distributed data. As multicloud becomes more and more accessible from cloud users, the use of MapReduce/Hadoop over multi-cloud is emerging; however, existing work has two issues in principle. First, it mainly focuses on maximizing throughput by improving data locality, but the perspective of cost optimization is missing. Second, conventional centralized optimization methods would not be able to scale well in multi-cloud environments due to its highly dynamic nature. We plan to solve the first issue by formalizing an optimization framework for MapReduce over multi-cloud including virtual machine and data transfer costs, and then the second issue by creating decentralized resource management middleware that considers multi-criteria (cost and performance) optimization. This paper reports progress we have made so far on these two directions.

show abstract

“…The following are few examples of applications that process geo-distributed datasets: climate science [8], [9], data generated by multinational companies [8], [10], [11], sensor networks [9], [12], stock exchanges [9], web crawling [13], [14], social networking applications [13], [14], biological data processing [8], [12], [15] such as DNA sequencing and human microbiome investigations, protein structure prediction, and molecular simulations, stream analysis [9], video feeds from distributed cameras, log files from distributed servers [12], geographical information systems (GIS) [4], and scientific applications [8], [9], [13], [16]. It should be noted down here that all the abovementioned applications generate a high volume of raw data across the globe; however, most analysis tasks require only a small amount of the original raw data for producing the final outputs or summaries [12].…”

mentioning

confidence: 99%

“…The main advantages of geo-distributed big-data processing are given in [33] and listed below: • A geo-distributed Hadoop/Spark-based system can perform data processing across nodes of multiple clusters while the standard Hadoop/Spark and their variants cannot process data at multiple clusters [33]. • More flexible services, e.g., resource sharing, load balancing, fault-tolerance, performance isolation, data isolation, and version isolation, can be achieved when a cluster is a part of a geo-distributed cluster [11], [16]. • A cluster can be scaled dynamically during the execution of a geo-distributed computation [33].…”

mentioning

confidence: 99%

A Survey on Geographically Distributed Big-Data Processing Using MapReduce

Dolev

Florissi

Gudes

et al. 2019

IEEE Trans. Big Data

View full text Add to dashboard Cite

Abstract-Hadoop and Spark are widely used distributed processing frameworks for large-scale data processing in an efficient and fault-tolerant manner on private or public clouds. These big-data processing systems are extensively used by many industries, e.g., Google, Facebook, and Amazon, for solving a large class of problems, e.g., search, clustering, log analysis, different types of join operations, matrix multiplication, pattern matching, and social network analysis. However, all these popular systems have a major drawback in terms of locally distributed computations, which prevent them in implementing geographically distributed data processing. The increasing amount of geographically distributed massive data is pushing industries and academia to rethink the current big-data processing systems. The novel frameworks, which will be beyond state-of-the-art architectures and technologies involved in the current system, are expected to process geographically distributed data at their locations without moving entire raw datasets to a single location. In this paper, we investigate and discuss challenges and requirements in designing geographically distributed data processing frameworks and protocols. We classify and study batch processing (MapReduce-based systems), stream processing (Spark-based systems), and SQL-style processing geo-distributed frameworks, models, and algorithms with their overhead issues.

show abstract

Improving Hadoop Service Provisioning in a Geographically Distributed Cloud

Abstract: Abstract-With

Cited by 25 publications

References 11 publications

Wide area analytics for geographically distributed datacenters

Wide area analytics for geographically distributed datacenters

Cost-Efficient High-Performance Internet-Scale Data Analytics over Multi-cloud Environments

A Survey on Geographically Distributed Big-Data Processing Using MapReduce

Contact Info

Product

Resources

About