Cost-Aware Big Data Processing Across Geo-Distributed Datacenters

Xiao, Wenhua; Bao, Weidong; Zhu, Xiaomin; Liu, Ling

doi:10.1109/tpds.2017.2708120

Cited by 48 publications

(24 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many existing studies [18], [19], [20] have dealt with the cost of running big data applications, such as social networks, and scientific computing [33]. These studies consider storage, computation and bandwidth to evaluate the cost of data geomigration and storage in distributed data-centers.…”

Section: B Interconnected Cloud Cost Modelsmentioning

confidence: 99%

“…It is composed of the cost of data transfer (outgoing network cost) from the partner clouds Cost ntwout,T , the local Internet bandwidth cost Cost bw,T corresponding to the bandwidth consumed locally to receive the previously outsourced objects, and the cost Cost wr,T to write the received objects on the local storage, see eq. (20).…”

Section: Cost Plcextmentioning

confidence: 99%

“…Cost bckmgr,T = Cost wr,T + Cost ntwout,T + Cost bw,T (20) The set of internal customers objects to be brought back O bmgr is deduced from the matrix A.…”

Section: Cost Plcextmentioning

confidence: 99%

“…Interconnected cloud cost models Fixed storage Traffic Geo-migration Heterogeneity Federation [17] x x [18], [19], [20] x x [21] x x x x [22], [23], [24] x [6] x x [25], [26], [24] x x x (b)…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Cost Model for Hybrid Storage Systems in a Cloud Federations

Chikhaoui

Boukhalfa

Boukhobza

2018

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

A cloud federation gives to cloud service providers (CSP) the opportunity to collaborate in order to offer a better QoS to customers at a lower cost. To do so, CSPs make some spare resources available to others at a reduced cost. One of the most critical resources is the storage system as it represents the main system bottleneck. From this point of view, how to efficiently place data in a federation of Clouds with heterogeneous storage systems is a real challenge. To address this issue, one needs to accurately estimate the data placement cost. In this paper, we propose a cost model for hybrid storage systems in a cloud federation for a Database as a Service (DBaaS) application. It takes into account the storage system characteristics, customers I/O workloads and SLA. The proposed cost model considers both 1) Internal customers data placement cost including local placement, outsourcing, back-migration and penalty costs, and 2) External customers data placement cost including insourcing and geo-migration costs. It can be used to help in the decision-making process which aims to enhance customers QoS and reduce CSPs costs in a federation. Simulation results showed the relevance of the considered costs. We have shown that mis-considering some sub-costs may lead to a 95% cost error for external customers data placement and 80% for outsourcing customers. This may cause significant financial loss.

show abstract

Section: B Interconnected Cloud Cost Modelsmentioning

confidence: 99%

Section: Cost Plcextmentioning

confidence: 99%

“…Cost bckmgr,T = Cost wr,T + Cost ntwout,T + Cost bw,T (20) The set of internal customers objects to be brought back O bmgr is deduced from the matrix A.…”

Section: Cost Plcextmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Cost Model for Hybrid Storage Systems in a Cloud Federations

Chikhaoui

Boukhalfa

Boukhobza

2018

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

show abstract

“…However, performance evaluation under cloud computing environment is not considered. In [16], a Big-Data computational model is adopted to reduce cost with the help of geo-distributed datacenters. This technique helps to decide the parameters to select the final data center.…”

mentioning

confidence: 99%

Optimized memory model for hadoop map reduce framework

Bhaskar¹,

Ranjan

2019

IJECE

View full text Add to dashboard Cite

Map Reduce is the preferred computing framework used in large data analysis and processing applications. Hadoop is a widely used Map Reduce framework across different community due to its open source nature. Cloud service provider such as Microsoft azure HDInsight offers resources to its customer and only pays for their use. However, the critical challenges of cloud service provider is to meet user task Service level agreement (SLA) requirement (task deadline). Currently, the onus is on client to compute the amount of resource required to run a job on cloud. This work present a novel memory optimization model for Hadoop Map Reduce framework namely MOHMR (Optimized Hadoop Map Reduce) to process data in real-time and utilize system resource efficiently. The MOHMR present accurate model to compute job memory optimization and also present a model to provision the amount of cloud resource required to meet task deadline. The MOHMR first build a profile for each job and computes memory optimization time of job using greedy approach. Experiment are conducted on Microsoft Azure HDInsight cloud platform considering different application such as text computing and bioinformatics application to evaluate performance of MOHMR of over existing model shows significant performance improvement in terms of computation time. Experiment are conducted on Microsoft Azure HDInsight cloud. Overall, good correlation is reported between practical memory optimization values and theoretical memory optimization values.

show abstract

Dynamic data replication and placement strategy in geographically distributed data centers

Bouhouch

Zbakh

Tadonki

2022

Concurrency and Computation

View full text Add to dashboard Cite

With the evolution of geographically distributed data centers in the Cloud Computing landscape along with the amount of data being processed in these data centers, which is growing at an exponential rate, processing massive data applications become an important topic. Since a given task may require many datasets for its execution and the datasets are spread over several different data centers, finding an efficient way to manage the datasets storage across nodes of a Cloud system is a difficult problem.In fact, the execution time of a task might be influenced by the cost of data transfers, which mainly depends on two criterias. The first one is the initial placement of the input datasets during the build-time phase, while the second is the replication of the datasets during the runtime phase. The replication is explicitly consider when datasets are being migrated over the data centers in order to make them locally available wherever needed. Data placement and data replication are important challenges in Cloud Computing. Nevertheless, many studies focus on data placement or data replication exclusively. In this paper, a combination of a data placement strategy followed by a dynamic data replication management strategy is proposed, with the purpose of reducing the associated cost of all data transfers between the (distant) data centers. Our proposed data placement approach considers the main characteristics of a data center such as storage capacity and read/write speeds to efficiently store the datasets, while our dynamic data replication management approach considers three parameters: the number of replicas in the system, the dependency between datasets and tasks and the storage capacity of data centers. The decision of when and whether to keep or to delete replicas is determined by the fulfillment of those three parameters. Our approach estimates the total execution time of the tasks as well as the monetary cost, considering the data transfers activity. Our experiments are conducted using Cloudsim simulator. The obtained results show that our proposed strategies produce an efficient data management by reducing the overheads of the data transfers, compared to both a data placement without replication (by 76%) and the selected data replication approach from Kouidri et al. (by 52%), and by improving the financial cost.

show abstract

Cost-Aware Big Data Processing Across Geo-Distributed Datacenters

Cited by 48 publications

References 25 publications

A Cost Model for Hybrid Storage Systems in a Cloud Federations

A Cost Model for Hybrid Storage Systems in a Cloud Federations

Optimized memory model for hadoop map reduce framework

Dynamic data replication and placement strategy in geographically distributed data centers

Contact Info

Product

Resources

About