An Energy Efficiency Optimization and Control Model for Hadoop Clusters

Wang, Haifeng; Cao, Yunpeng

doi:10.1109/access.2019.2907018

Cited by 13 publications

(5 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While Figure 1a is obtained by adding two real operational data points to the initial synthetic data set, Figure 1b is obtained by adding five operational data points. 4 Importing a larger number of operational data points, one would expect that the prediction curve gets closer to the expected values curve. However, comparing the two graphs, it is evident that adding new points, instead of enhancing the ML model prediction, actually makes the ML output worse.…”

Section: Background and Motivationsmentioning

confidence: 99%

“…Most of big data applications implemented today are based on the MapReduce (MR) programming model, which is the most common adopted solution. MapReduce-based systems have risen as a scalable and cost effective solution for massively parallel data processing [3] The majority of large-scale data intensive applications designed by MR model are deployed and executed on a large-scale distributed Hadoop system [4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Hybrid Machine Learning Approach for Performance Modeling of Cloud-Based Big Data Applications

Ataie

Evangelinou

Gianniti

et al. 2021

The Computer Journal

View full text Add to dashboard Cite

Nowadays, Apache Hadoop and Apache Spark are two of the most prominent distributed solutions for processing big data applications on the market. Since in many cases these frameworks are adopted to support business critical activities, it is often important to predict with fair confidence the execution time of submitted applications, for instance when service-level agreements are established with end-users. In this work, we propose and validate a hybrid approach for the performance prediction of big data applications running on clouds, which exploits both analytical modeling and machine learning (ML) techniques and it is able to achieve a good accuracy without too many time consuming and costly experiments on a real setup. The experimental results show how the proposed approach attains improvement in accuracy, number of experiments to be run on the operational system and cost over applying ML techniques without any support from analytical models. Moreover, we compare our approach with Ernest, an ML-based technique proposed in the literature by the Spark inventors. Experiments show that Ernest can accurately estimate the performance in interpolating scenarios while it fails to predict the performance when configurations with increasing number of cores are considered. Finally, a comparison with a similar hybrid approach proposed in the literature demonstrates how our approach significantly reduce prediction errors especially when few experiments on the real system are performed.

show abstract

Section: Background and Motivationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Hybrid Machine Learning Approach for Performance Modeling of Cloud-Based Big Data Applications

Ataie

Evangelinou

Gianniti

et al. 2021

The Computer Journal

View full text Add to dashboard Cite

show abstract

“…Wang and Cao [23] also supported the use of naïve Bayes in their clinical experiments to understand the neuroimaging framework of humans. The research involves different stimulus phases, but the data collection and classification presented a highly intelligent insight into how data classification can be managed if performed correctly.…”

Section: Classification Approachesmentioning

confidence: 88%

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Kumar,

Varshney,

Bhatiya

et al. 2023

Big Data Min. Anal.

View full text Add to dashboard Cite

We live in an age where everything around us is being created. Data generation rates are so scary, creating pressure to implement costly and straightforward data storage and recovery processes. MapReduce model functionality is used for creating a cluster parallel, distributed algorithm, and large datasets. The MapReduce strategy from Hadoop helps develop a community of non-commercial use to offer a new algorithm for resolving such problems for commercial applications as expected from this working algorithm with insights as a result of disproportionate or discriminatory Hadoop cluster results. Expected results are obtained in the work and the exam conducted under this job; many of them are scheduled to set schedules, match matrices' data positions, clustering before determining to click, and accurate mapping and internal reliability to be closed together to avoid running and execution times. Mapper output and proponents have been implemented, and the map has been used to reduce the function. The execution input key/value pair and output key/value pair have been set. This paper focuses on evaluating this technique for the efficient retrieval of large volumes of data. The technique allows for capabilities to inform a massive database of information, from storage and indexing techniques to the distribution of queries, scalability, and performance in heterogeneous environments. The results show that the proposed work reduces the data processing time by 30%.

show abstract

“…In this subsection the real-world dataset was used to evaluate performance of HDDM. This real-world workload is from real-time computing by taxi trajectories datasets generated by 33000 taxis over a period of six months [29,30]. The taxi trajectory is a sequence of GPS points pertaining to a trip.…”

Section: E Real-world Workloads Analysismentioning

confidence: 99%

Virtual Cluster Deployment Model for Large-Scale Data Processing Jobs

Cao

Wang

2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

The virtual cluster needs to deploy virtual nodes with different computing modes for Large-scale data computing jobs including many tasks with different computing modes. There are resource unbalance among the virtual cluster subsets, which is consisted of virtual nodes with the same computing mode. So the resource utilization of virtual cluster is poor due to the time-varying and complexity of workloads. To solve these issues, a novel dynamic deployment model based on Docker was proposed to optimize resource configuration of virtual cluster. Hybrid deployment scheme includes coarse-grained deployment and fine-grained deployment in order to absorb the mutation of resource demand when processing the jobs encapsulated with complex computing logic.This deployment model can reduce job waiting time and improve performance by providing more resource for running jobs. The coarse-grained deployment mechanism has the ability of decreasing deploying overhead and improving stability from global optimal perspective. And the fine-grained deployment mechanism can increases the deploying accuracy from local optimal perspective. The experiments show that this model can improve the execution efficiency of job by 2.3 percent compared with static deploying scheme and improve the virtual CPU utilization about 7.9 to 25.9 percent compared with other approaches.

show abstract

An Energy Efficiency Optimization and Control Model for Hadoop Clusters

Cited by 13 publications

References 39 publications

A Hybrid Machine Learning Approach for Performance Modeling of Cloud-Based Big Data Applications

A Hybrid Machine Learning Approach for Performance Modeling of Cloud-Based Big Data Applications

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Virtual Cluster Deployment Model for Large-Scale Data Processing Jobs

Contact Info

Product

Resources

About