Cost-Effective Cloud Server Provisioning for Predictable Performance of Big Data Analytics

Xu, Fei; Zheng, Haoyue; Jiang, Hangyi; Shao, Wujie; Liu, Haikun; Zhou, Zhi

doi:10.1109/tpds.2018.2873397

Cited by 40 publications

(18 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cloud infrastructure is scalable and it consists of large number with huge sizes (operation capabilities) of physical computing devices. Such huge infrastructure is capable of providing high computation in fraction of time, even for the analysis of huge data [42]. The power demand of such data centers is 1.5% of world's total demand with annual growth rate at 4.3% [27].…”

Section: Related Workmentioning

confidence: 99%

Cost Efficient Real Time Electricity Management Services for Green Community Using Fog

et al. 2020

View full text Add to dashboard Cite

The computing devices in data centers of cloud and fog remain in continues running cycle to provide services. The long execution state of large number of computing devices consumes a significant amount of power, which emits an equivalent amount of heat in the environment. The performance of the devices is compromised in heating environment. The high powered cooling systems are installed to cool the data centers. Accordingly, data centers demand high electricity for computing devices and cooling systems. Moreover, in Smart Grid (SG) managing energy consumption to reduce the electricity cost for consumers and minimum rely on fossil fuel based power supply (utility) is an interesting domain for researchers. The SG applications are time-sensitive. In this paper, fog based model is proposed for a community to ensure real-time energy management service provision. Three scenarios are implemented to analyze cost efficient energy management for power-users. In first scenario, community’s and fog’s power demand is fulfilled from the utility. In second scenario, community’s Renewable Energy Resources (RES) based Microgrid (MG) is integrated with the utility to meet the demand. In third scenario, the demand is fulfilled by integrating fog’s MG, community’s MG and the utility. In the scenarios, the energy demand of fog is evaluated with proposed mechanism. The required amount of energy to run computing devices against number of requests and amount of power require cooling down the devices are calculated to find energy demand by fog’s data center. The simulations of case studies show that the energy cost to meet the demand of the community and fog’s data center in third scenario is 15.09% and 1.2% more efficient as compared to first and second scenarios, respectively. In this paper, an energy contract is also proposed that ensures the participation of all power generating stakeholders. The results advocate the cost efficiency of proposed contract as compared to third scenario. The integration of RES reduce the energy cost and reduce emission of CO 2 . The simulations for energy management and plots of results are performed in Matlab. The simulation for fog’s resource management, measuring processing, and response time are performed in CloudAnalyst.

show abstract

Section: Related Workmentioning

confidence: 99%

Cost Efficient Real Time Electricity Management Services for Green Community Using Fog

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The many details of the Spark framework, its scheduler, and the other parameters of the cluster require advanced mathematical models to accurately predict the job completion time of a workload. The three main approaches in existing literature on the topic are simulation [10], [11], machine learning [12], [13], [14], [15], and analytical modeling [16], [17], [18]. Efforts to simulate a Spark cluster have resulted in a number of comprehensive simulators that mimic the actual framework's behavior [10].…”

Section: Introductionmentioning

confidence: 99%

Fixed-Point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

Karimian-Aliabadi

Aseman-Manzar

Entezari‐Maleki

et al. 2023

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

Companies depend on mining data to grow their business more than ever. To achieve optimal performance of Big Data analytics workloads, a careful configuration of the cluster and the employed software framework is required. The lack of flexible and accurate performance models, however, render this a challenging task. This paper fills this gap by presenting accurate performance prediction models based on Stochastic Activity Networks (SANs). In contrast to existing work, the presented models consider multiple work queues, a critical feature to achieve high accuracy in realistic usage scenarios. We first introduce a monolithic analytical model for a multi-queue YARN cluster running DAG-based Big Data applications that models each queue individually. To overcome the limited scalability of the monolithic model, we then present a fixed-point model that iteratively computes the throughput of a single queue with respect to the rest of the system until a fixed-point is reached. The models are evaluated on a real-world cluster running the widely-used Apache Spark framework and the YARN scheduler. Experiments with the common transaction-based TPC-DS benchmark show that the proposed models achieve an average error of only 5.6% in predicting the execution time of the Spark jobs. The presented models enable businesses to optimize their cluster configuration for a given workload and thus to reduce their expenses and minimize service level agreement (SLA) violations. Makespan minimization and per-stage analysis are examined as representative efforts to further assess the applicability of our proposition.

show abstract

“…The traditional data processing method is to transmit the data to a centralized cloud server for calculation. The cloud server has strong computing power and can handle tasks with a large amount of calculation [4,5]. However, as the number of IoT devices increases, the unified upload of computing tasks to cloud servers for computing will cause network data congestion.…”

Section: Introductionmentioning

confidence: 99%

Task offloading and resource allocation algorithm based on mobile edge computing in Internet of Things environment

Liu

2021

The Journal of Engineering

View full text Add to dashboard Cite

This paper proposes a task offloading and resource allocation algorithm based on mobile edge computing. Firstly, for the long-term performance optimization of small base stations, the system model is established according to task arrival characteristics, credit relationship between small base stations, time delay and energy consumption of computing tasks and cable channel congestion. Secondly, the energy consumption deficit queue based on Lyapunov drift penalty technology used for the energy consumption constraint of small base stations in long-term optimization process. The energy consumption deficit queue is established for each small base station to couple the energy consumption and time of small base stations, so that small base stations can meet the energy consumption constraints in longterm optimization process. Finally, game theory is introduced to calculate offloading weight by the offloading weight model based on Shapley value. Besides, the offloading weight is calculated equitably according to the return of different tasks. Simulation results on MAT-LAB platform show that the proposed algorithm can achieve Nash equilibrium after finite iterations. Moreover, its performance on energy consumption, time delay and number of tasks successfully offloaded is better than other comparison strategies.This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Cost-Effective Cloud Server Provisioning for Predictable Performance of Big Data Analytics

Cited by 40 publications

References 34 publications

Cost Efficient Real Time Electricity Management Services for Green Community Using Fog

Cost Efficient Real Time Electricity Management Services for Green Community Using Fog

Fixed-Point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

Task offloading and resource allocation algorithm based on mobile edge computing in Internet of Things environment

Contact Info

Product

Resources

About