A computing resources prediction approach based on ensemble learning for complex system simulation in cloud environment

Wang, Shuai; Zhu, Feng; Yao, Yiping; Tang, Wenjie; Xiao, Yuhao; Xiong, Siqi

doi:10.1016/j.simpat.2020.102202

Cited by 17 publications

(5 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The historical trace (HT) model predicts the required resources based on the nature of jobs and their execution patterns. However, some researchers have claimed that one cannot accurately predict the resources needed for running applications in an HPC environment [2], [3], [5], [40], [43]. These claims motivate us to work on dependency prediction of long-term resource utilization.…”

Section: A Motivationmentioning

confidence: 99%

“…We have chosen normal and joint distribution for the above equation [eq (2)] because the mutual information will be the intermediate part of the information from both individuals. So, the normalized mutual information will be written as [eq(3)]:…”

Section: A Overview Of Ensemble Algorithmmentioning

confidence: 99%

“…The computation power of supercomputers brings a different perspective to the research and overcomes many traditional computing limitations [2], [6], [8]. Examples of such limitations are non-linear relationship modeling, solving random problems, solving long-iterative social evolution problems and algorithms, etc.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dependency Prediction of Long-Time Resource Uses in HPC Environment

Upadhyay,

Singh,

Dwivedi

2023

IEEE Access

View full text Add to dashboard Cite

High-Performance computing provides a new infrastructure for scientific calculation and its simulation. However, unbalanced load distribution among the processors causes a decreased performance in such computations, and creates a massive requirement of computing resource allocation, that requires an increased simulation. Therefore long-range resource utilization prediction becomes essential to achieve optimal performance in an HPC environment. This paper introduces a novel ensemble technique, which includes two algorithms, the Feature-based capability prediction algorithm(FBCA), and the Accuracy and Relative Runtime Error Prediction Algorithm (ARRE). A three-level architectural framework (the simulation environment, resource prediction, and resource queue) has also been proposed and tested on Phold and SoS. The proposed framework can deal with the requirements of computing and simulations. The FBCA algorithm reduces the redundancy between available features, and the ARRE algorithm ensures our ensemble technique's effectiveness. We have compared the performance of the proposed schemes with other existing methods such as the Regressive Approach, Linear Regression and Random Forest, and found that our proposed algorithm achieves better accuracy from 8% to 18%.

show abstract

Section: A Motivationmentioning

confidence: 99%

Section: A Overview Of Ensemble Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Dependency Prediction of Long-Time Resource Uses in HPC Environment

Upadhyay,

Singh,

Dwivedi

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…We used support vector regression (SVR) [25], RF [26], back propagation neural network (BPNN) [27], and MS-GEP to model the map-reduce job CP-EE, respectively. SVR, RF and BPNN were trained and tested on the same datasets as MS-GEP.…”

Section: Plos Onementioning

confidence: 99%

An improved gene expression programming algorithm for function mining of map-reduce job execution in catenary monitoring systems

Ding,

Jiang,

Tan

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

Gene expression programming (GEP) is one of the most prominent algorithms in function mining. In order to obtain a more accurate function model in configuration parameters-execution efficiency (CP-EE) of map-reduce job in the high-speed railway catenary monitoring system, this paper proposes a novel algorithm, called GEP based on multi-strategy (MS-GEP). Compared to traditional GEP, the proposed algorithm can escape premature convergence and jump out of local optimum. First, an adaptive mutation rate is designed according to the evolutionary generations, population diversity, and individual fitness values. A manual intervention strategy is then proposed to determine whether the algorithm enters the dilemma of local optimum based on the generations of population evolutionary stagnation. Finally, the average quality of the population is changed by randomly replacing individuals, and the ancestral population is traced to change the evolutionary direction. The experimental results on the benchmarks of function mining show that the proposed MS-GEP has better solution quality and higher population diversity than other GEP algorithms. Furthermore, the proposed MS-GEP has higher accuracy on the function model of CP-EE of high-speed railway catenary monitoring system than other commonly used algorithms in the field of function mining.

show abstract

“…The dynamic assessment of risks should be taken place in order to migrate the services between the worker nodes without affecting the performance of scheduling. After the risk assessment the resource should be given to the legitimate queries by discovering the resources [20].…”

Section: Introductionmentioning

confidence: 99%

SparkGrid: Blockchain Assisted Secure Query Scheduling and Dynamic Risk Assessment for Live Migration of Services in Apache Spark based Grid Environment

G.M.

Nalini

2022

Preprint

View full text Add to dashboard Cite

Grid computing is an emerging technology that enabled the heterogeneous collection of data and provisioning of services to the users. Due to the high amount of incoming heterogeneous request, grid computing needs an efficient scheduling to reduce execution time and satisfy Service Level Agreement (SLA) and Quality of Service (QoS) requirements. For that purpose, we proposed SprakGrid method to reduce execution time and satisfying SLA, QoS requirements. The proposed work includes four consecutive phases which are explained as follows, in first we perform user authentication in order to ensure the legitimacy of the users using Elliptic Curve based Chaos Theory (ECCT) algorithm which generate secret key and stored it into the blockchain. In second we perform query scheduling for resource discovery using Soft Actor Critic (SAC) algorithm by considering 3P’s parameters which is performed by spark environment that schedules optimal resources based on the service request. In third, we perform risk assessment and request dropping, in which the risk nodes of workers are evaluated by master node. To address the resource wastage by attacker, this research evaluates the risk value in a dynamic manner using Shannon entropy. Based on the risk assessment the requestsare classified into two classes such as normal and malicious. In fourth we perform service live migration, in which the malicious requests are dropped and normal request are migrated from source node to target node using Multi-Constraints based Emperor Penguin Optimization (MC-EPO). Finally, simulation is performed by GridSim and the simulation results demonstrate that the proposed SparkGrid method achieves superior performance compared to other state-of-the art methods.

show abstract

A computing resources prediction approach based on ensemble learning for complex system simulation in cloud environment

Cited by 17 publications

References 26 publications

Dependency Prediction of Long-Time Resource Uses in HPC Environment

Dependency Prediction of Long-Time Resource Uses in HPC Environment

An improved gene expression programming algorithm for function mining of map-reduce job execution in catenary monitoring systems

SparkGrid: Blockchain Assisted Secure Query Scheduling and Dynamic Risk Assessment for Live Migration of Services in Apache Spark based Grid Environment

Contact Info

Product

Resources

About