Building Semi-Elastic Virtual Clusters for Cost-Effective HPC Cloud Resource Provisioning

Niu, Shuangcheng; Zhai, Jidong; Ma, Xiaosong; Tang, Xiongchao; Chen, Wenguang; Zheng, Weimin

doi:10.1109/tpds.2015.2476459

Cited by 25 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So, the HPC applications can take advantages of these two models. [14]- [18]. For example in [18], the authors presented a new approach for horizontally scaling cloud resources, and in [19] for load balancing, and resources scheduling [20] in cloud.…”

Section: B Resultsmentioning

confidence: 99%

Toward a New Massively Distributed Virtual Machine based Cloud Micro-Services Team Model for HPC: SPMD Applications

Benchara¹,

Youssfi²,

Bouattane³

et al. 2017

ijacsa

View full text Add to dashboard Cite

Abstract-This paper aims to propose a new massively distributed virtual machine with scalable and efficient parallel computing models for High Performance Computing (HPC). The message passing paradigm of the Processing Units has a significant impact on HPC with high communication cost that penalizes the performance of these models. Accordingly, the proposed micro-services model allows the HPC applications to enhance the processing power with low communication cost. Thus, the model based Micro-services Virtual Processing Units (MsVPUs) cooperate using asynchronous communication mechanism through the Advanced Message Queuing Protocol (AMQP) protocol in order to maintain the scalability of the Single Program Multiple Data (SPMD) applications. Additionally, this mechanism enhances also the efficiency of the model based load balancing service with time optimized load balancing strategy. The proposed virtual machine is tested and validated through an application of fine grained parallel programs for big data classification. Experimental results present reduced execution time compared to the virtual machine based mobile agent's model.

show abstract

Section: B Resultsmentioning

confidence: 99%

Toward a New Massively Distributed Virtual Machine based Cloud Micro-Services Team Model for HPC: SPMD Applications

Benchara¹,

Youssfi²,

Bouattane³

et al. 2017

ijacsa

View full text Add to dashboard Cite

show abstract

“…Intensive research work has been made on the cost-effective computation in the cloud environment. A Semi-Elastic Cluster (SEC) computing model [32] has been proposed for organizations to reserve and dynamically resize a virtual cloud-based cluster. The race-driven results show that such a model has a 61% percent cost saving than individual users acquiring and managing cloud resources without causing longer average job wait time.…”

Section: Related Workmentioning

confidence: 99%

Cutting the Unnecessary Long Tail: Cost-Effective Big Data Clustering in the Cloud

Wang

Gao³

et al. 2022

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

Clustering big data often requires tremendous computational resources where cloud computing is undoubtedly one of the promising solutions. However, the computation cost in the cloud can be unexpectedly high if it cannot be managed properly. The long tail phenomenon has been observed widely in the big data clustering area, which indicates that the majority of time is often consumed in the middle to late stages in the clustering process. In this research, we try to cut the unnecessary long tail in the clustering process to achieve a sufficiently satisfactory accuracy at the lowest possible computation cost. A novel approach is proposed to achieve cost-effective big data clustering in the cloud. By training the regression model with the sampling data, we can make widely used k-means and EM (Expectation-Maximization) algorithms stop automatically at an early point when the desired accuracy is obtained. Experiments are conducted on four popular data sets and the results demonstrate that both k-means and EM algorithms can achieve high cost-effectiveness in the cloud with our proposed approach. For example, in the case studies with the much more efficient k-means algorithm, we find that achieving a 99% accuracy needs only 47.71%-71.14% of the computation cost required for achieving a 100% accuracy while the less efficient EM algorithm needs 16.69%-32.04% of the computation cost. To put that into perspective, in the United States land use classification example, our approach can save up to $94,687.49 for the government in each use.

show abstract

“…A semi-elastic cluster computing model [25] was introduced for organizations to reserve and dynamically adjust the size of cloud-based virtual clusters. The experiment results indicated that such a model can save more than 60% cost than individual users acquiring and managing cloud resources without leading to longer average job wait times.…”

Section: Cost-effective Cloud Computingmentioning

confidence: 99%

Cost-effective Land Cover Classification for Remote Sensing Images

Li,

Wang,

et al. 2021

Preprint

View full text Add to dashboard Cite

Land cover maps are of vital importance to various fields such as land use policy development, ecosystem services, urban planning and agriculture monitoring, which are mainly generated from remote sensing image classification techniques. Traditional land cover classification usually needs tremendous computational resources, which often becomes a huge burden to the remote sensing community. Undoubtedly cloud computing is one of the best choices for land cover classification, however, if not managed properly, the computation cost on the cloud could be surprisingly high. Recently, cutting the unnecessary computation long tail has become a promising solution for saving the cost in the cloud. For land cover classification, it is generally not necessary to achieve the best accuracy and 85% can be regarded as a reliable land cover classification. Therefore, in this paper, we propose a framework for cost-effective remote sensing classification. Given the desired accuracy, the clustering algorithm can stop early for cost-saving whilst achieving sufficient accuracy for land cover image classification. Experimental results show that achieving 85%-99.9% accuracy needs only 27.34%-60.83% of the total cloud computation cost for achieving a 100% accuracy. To put it into perspective, for the US land cover classification example, the proposed approach can save over $721,580.46 for the government in each single-use when the desired accuracy is 90%.

show abstract

Building Semi-Elastic Virtual Clusters for Cost-Effective HPC Cloud Resource Provisioning

Cited by 25 publications

References 25 publications

Toward a New Massively Distributed Virtual Machine based Cloud Micro-Services Team Model for HPC: SPMD Applications

Toward a New Massively Distributed Virtual Machine based Cloud Micro-Services Team Model for HPC: SPMD Applications

Cutting the Unnecessary Long Tail: Cost-Effective Big Data Clustering in the Cloud

Cost-effective Land Cover Classification for Remote Sensing Images

Contact Info

Product

Resources

About