High Performance Computing in the cloud: Deployment, performance and cost efficiency

Abstract

High-Performance Computing (HPC) in the cloud has reached the mainstream and is currently a hot topic in the research community and the industry. The attractiveness of cloud for HPC is the capability to run large applications on powerful, scalable hardware without needing to actually own or maintain this hardware. In this paper, we conduct a detailed comparison of HPC applications running on three cloud providers, Amazon EC2, Microsoft Azure and Rackspace. We analyze three important characteristics of HPC, deployment facilities, performance and cost efficiency and compare them to a cluster of machines.For the experiments, we used the well-known NAS parallel benchmarks as an example of general scientific HPC applications to examine the computational and communication performance. Our results show that HPC applications can run efficiently on the cloud. However, care must be taken when choosing the provider, as the differences between them are large. The best cloud provider depends on the type and behavior of the application, as well as the intended usage scenario. Furthermore, our results show that HPC in the cloud can have a higher performance and cost efficiency than a traditional cluster, up to 27% and 41%, respectively. C OpenMP, MPI LU Lower and Upper Triangular Regular communication Fortran OpenMP, MPI MG Multigrid Regular communication Fortran OpenMP, MPI SP Scalar Pentadiagonal Floating point performance Fortran OpenMP, MPI UA Unstructured Adaptive Irregular communication Fortran OpenMP B. Machines

Leveraging Cloud Heterogeneity for Cost-Efficient Execution of Parallel Applications

“…In the figures, the line represents the cost efficiency when varying the mix of instance types. To calculate the cost efficiency metric, we use the following equation [13]. Several interesting results can be pointed out.…”

“…Public cloud providers such as Amazon's EC2 and Microsoft's Azure provide a large number of cloud instance types with different numbers of cores, processing speeds, and network interconnections [18]. Research in this area focuses mostly on porting applications to the cloud [12], evaluating their performance and cost efficiency [13,18], and improving communication performance [4,3,2].…”

Job placement advisor based on turnaround predictions for HPC hybrid clouds

“…We empirically identified a power law dependency between speed and uncertainty 6. In this work we decided to ignore the runtime uncertainty, as the major source of uncertainty comes from wait time predictions.…”

“…cloud 6 . Intuitively, it tests whether the predicted total time for executing a job in the local cluster (w p + r p ) exceeds the time of running the same job in the cloud (r cloud (j, r p )).…”

“…Current work on HPC cloud, also known as HPCaaS (HPC as a Service) [1], has mainly focused on understanding the cost-benefits of using cloud over on-premise clusters [2][3][4][5][6][7][8]. In addition, the aim has been to evaluate the performance gap between cloud and on-premise resources [9][10][11][12].…”