Machine Learning with Sensitivity Analysis to Determine Key Factors Contributing to Energy Consumption in Cloud Data Centers

Foo, Yong Wee; Goh, Cindy; Li, Yun

doi:10.1109/icccri.2016.24

Cited by 11 publications

(13 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Foo et al [32–34] develop an EC prediction model based on an evolutionary NN combining with several novel mechanisms of a genetic algorithm. The results, both in terms of forecasting speed and accuracy, suggest that the evolutionary NN approach to EC forecasting for cloud computing is highly promising.…”

Section: Related Workmentioning

confidence: 99%

Power consumption model based on feature selection and deep learning in cloud computing scenarios

Yang

2020

IET Communications

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Power consumption model based on feature selection and deep learning in cloud computing scenarios

Yang

2020

IET Communications

View full text Add to dashboard Cite

“…[17], a series of seven different sensitivity analysis methods were reviewed. As we show in the next section, our approach can be related to the weights method [18] . The difference between our method and the classical weights method is that, in our approach, the connection weight among the nodes is provided by the error covariance matrices computed in the DA process.…”

Section: Related Work and Contribution Of The Present Studymentioning

confidence: 99%

“…1 C e x k C1 (18) where M D OEm 1 ; m 2 is the amplitude vector which can be adjusted according to the reality. To achieve the model simulation, we design a cascade controller to avoid the system from divergence.…”

Section: Double-integral Mass Dot Systemmentioning

confidence: 99%

Model error correction in data assimilation by integrating neural networks

Zhu

Arcucci

et al. 2019

Big Data Min. Anal.

View full text Add to dashboard Cite

In this paper, we suggest a new methodology which combines Neural Networks (NN) into Data Assimilation (DA). Focusing on the structural model uncertainty, we propose a framework for integration NN with the physical models by DA algorithms, to improve both the assimilation process and the forecasting results. The NNs are iteratively trained as observational data is updated. The main DA models used here are the Kalman filter and the variational approaches. The effectiveness of the proposed algorithm is validated by examples and by a sensitivity study.

show abstract

“…It is a multidisciplinary field of science incorporating computer science, statistics, information theory and artificial intelligence [59]. Machine learning has been used to model: energy utilization of clusters [60] and performance prediction of jobs [61]. There is also some work reported to optimize the use of resources utilization [62], but the potential of using machine learning to model characteristics of Hadoop job for diverse system resources is still largely an un-explored area of research.…”

Section: Modelling Resource and Performance Based On Job-characteristicsmentioning

confidence: 99%

“…The inter-related impact of these attributes can be hard to predict manually. Automated procedures such as machine learning can be effectively used to model such complex relationship [60]. Such a correlation between attributes can be useful to cluster designers and managers to optimize its performance and resource utilisation [61] [62].…”

Section: Research Goal and Objectivesmentioning

confidence: 99%

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Tariq¹

View full text Add to dashboard Cite

<p>There is a huge and rapidly increasing amount of data being generated by social media, mobile applications and sensing devices. Big data is the term usually used to describe such data and is described in terms of the 3Vs - volume, variety and velocity. In order to process and mine such a massive amount of data, several approaches and platforms have been developed such as Hadoop. Hadoop is a popular open source distributed and parallel computing framework. It has a large number of configurable parameters which can be set before the execution of jobs to optimize the resource utilization and execution time of the clusters. These parameters have a significant impact on system resources and execution time. Optimizing the performance of a Hadoop cluster by tuning such a large number of parameters is a tedious task. Most current big data modeling approaches do not include the complex interaction between configuration parameters and the cluster environment changes such as use of different datasets or types of query. This makes it difficult to predict for example the execution time of a job or resource utilization of a cluster. Other attributes include configuration parameters, the structure of query, the dataset, number of nodes and the infrastructure used. Our first main objective was to design reliable experiments to understand the relationship between attributes. Before designing and implementing the actual experiment we applied Hazard and Operability (HAZOP) analysis to identify operational hazards. These hazards can affect normal working of cluster and execution of Hadoop jobs. This brainstorming activity improved the design and implementation of our experiments by improving the internal validity of the experiments. It also helped us to identify the considerations that must be taken into account for reliable results. After implementing our design, we characterized the relationship between different Hadoop configuration parameters, network and system performance measures. Our second main objective was to investigate the use of machine learning to model and predict the resource utilization and execution time of Hadoop jobs. Resource utilization and execution time of Hadoop jobs are affected by different attributes such as configuration parameters and structure of query. In order to estimate or predict either qualitatively or quantitatively the level of resource utilization and execution time, it is important to understand the impact of different combinations of these Hadoop job attributes. You could conduct experiments with many different combinations of parameters to uncover this but it is very difficult to run such a large number of jobs with different combinations of Hadoop job attributes and then interpret the data manually. It is very difficult to extract patterns from the data and give a model that can generalize for an unseen scenario. In order to automate the process of data extraction and modeling the complex behavior of different attributes of Hadoop job machine learning was used. Our decision tree based approach enabled us to systematically discover significant patterns in data. Our results showed that the decision tree models constructed for different resources and execution time were informative and robust. They were able to generalize over a wide range of minor and major environmental changes such as change in dataset, cluster size and infrastructure such as Amazon EC2. Moreover, the use of different correlation and regression techniques, such as M5P, Pearson's correlation and k-means clustering, confirmed our findings and provided further insight into the relationship of different attributes and with each other. M5P is a classification and regression technique that predicted the functional relationships among different job attributes. The use of k-means clustering allowed us to see the experimental runs that shows similar resource utilization and execution time. Statistical significance tests, were used to validate the significance of changes in results of different experimental runs, also showed the effectiveness of our resource and performance modelling and prediction method.</p>

show abstract

Machine Learning with Sensitivity Analysis to Determine Key Factors Contributing to Energy Consumption in Cloud Data Centers

Cited by 11 publications

References 15 publications

Power consumption model based on feature selection and deep learning in cloud computing scenarios

Power consumption model based on feature selection and deep learning in cloud computing scenarios

Model error correction in data assimilation by integrating neural networks

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Contact Info

Product

Resources

About