A hybrid HPC/cloud distributed infrastructure: Coupling EC2 cloud resources with HPC clusters to run large tightly coupled multiscale applications

Belgacem, Mohamed Ben; Chopard, Bastien

doi:10.1016/j.future.2014.08.003

Cited by 32 publications

(18 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our opinion, this is the only sensible way to fully exploit all the intricacies of state-of-the-art high end computing resources for multiscale simulations. We have already demonstrated the benefits in distributed computing environments (distributed multiscale computing) [35][36][37], including clouds [79].…”

Section: Discussionmentioning

confidence: 99%

Multiscale computing in the exascale era

Alowayyed

Groen

Coveney

et al. 2017

Journal of Computational Science

View full text Add to dashboard Cite

We expect that multiscale simulations will be one of the main high performance computing workloads in the exascale era. We propose multiscale computing patterns as a generic vehicle to realise load balanced, fault tolerant and energy aware high performance multiscale computing. Multiscale computing patterns should lead to a separation of concerns, whereby application developers can compose multiscale models and execute multiscale simulations, while pattern software realises optimized, fault tolerant and energy aware multiscale computing. We introduce three multiscale computing patterns, present an example of the extreme scaling pattern, and discuss our vision of how this may shape multiscale computing in the exascale era.

show abstract

Section: Discussionmentioning

confidence: 99%

Multiscale computing in the exascale era

Alowayyed

Groen

Coveney

et al. 2017

Journal of Computational Science

View full text Add to dashboard Cite

show abstract

“…One can find in the literature several exercises of integration of hybrid infrastructures, integrating cloud resources, commercial or private, with High Performance Computing (HPC) resources. Those efforts focus on outbursting resources from the cloud, when the HPC system does not provide enough resources to solve a particular problem [41].…”

Section: High Performance Computing Integrationmentioning

confidence: 99%

An efficient cloud scheduler design supporting preemptible instances

García

Fernández-del-Castillo

Plasencia

2019

Future Generation Computer Systems

View full text Add to dashboard Cite

Maximizing resource utilization by performing an efficient resource provisioning is a key factor for any cloud provider: commercial actors can maximize their revenues, whereas scientific and non-commercial providers can maximize their infrastructure utilization. Traditionally, batch systems have allowed data centers to fill their resources as much as possible by using backfilling and similar techniques. However, in an IaaS cloud, where virtual machines are supposed to live indefinitely, or at least as long as the user is able to pay for them, these policies are not easily implementable. In this work we present a new scheduling algorithm for IaaS providers that is able to support preemptible instances, that can be stopped by higher priority requests without introducing large modifications in the current cloud schedulers. This scheduler enables the implementation of new cloud usage and payment models that allow more efficient usage of the resources and potential new revenue sources for commercial providers. We also study the correctness and the performace overhead of the proposed scheduler agains existing solutions.

show abstract

“…Belgacem and Chopard [18] reported their experience of porting a parallel 3D simulation in the computational fluid dynamics domain to a hybrid cloud consisting of a cluster located in Switzerland HPC Cloud for Scientific and Business Applications 8:13 Table 3. Overview of the related work on performance optimization for HPC cloud.…”

Section: Usability: User Interaction and High-level Servicesmentioning

confidence: 99%

HPC Cloud for Scientific and Business Applications

et al. 2018

View full text Add to dashboard Cite

High Performance Computing (HPC) clouds are becoming an alternative to on-premise clusters for executing scientific applications and business analytics services. Most research efforts in HPC cloud aim to understand the cost-benefit of moving resource-intensive applications from on-premise environments to public cloud platforms. Industry trends show hybrid environments are the natural path to get the best of the on-premise and cloud resources-steady (and sensitive) workloads can run on on-premise resources and peak demand can leverage remote resources in a pay-as-you-go manner. Nevertheless, there are plenty of questions to be answered in HPC cloud, which range from how to extract the best performance of an unknown underlying platform to what services are essential to make its usage easier. Moreover, the discussion on the right pricing and contractual models to fit small and large users is relevant for the sustainability of HPC clouds. This paper brings a survey and taxonomy of efforts in HPC cloud and a vision on what we believe is ahead of us, including a set of research challenges that, once tackled, can help advance businesses and scientific discoveries. This becomes particularly relevant due to the fast increasing wave of new HPC applications coming from big data and artificial intelligence. users have no visibility or concerns on costs of running jobs. However, large clusters do incur expenses and, when not properly managed, can generate resource wastage and poor quality of service.Motivated by the different utilization levels of clusters around the globe and by the need to run even larger parallel programs, in the early 2000s, Grid Computing became relevant for the HPC community. Grids offer users access to powerful resources managed by autonomous administrative domains [50,51]. The notion of monetary costs for running applications was soft, favoring a more collaborative model of resource sharing. Therefore, quality of service was not strict in Grids, having users relying on best-effort policies to run applications.In the late 2000s, cloud computing [8,26,91] was quickly increasing its maturity level and popularity, and studies started to emerge on the viability of executing HPC applications on remote cloud resources. These applications, which consume more resources than traditional cloud applications and usually are executed in batches rather than 24x7 services, range from parallel applications written in Message Passing Interface (MPI) [58,59] to the newest big data [11,14,39,101] and artificial intelligence applications-the latter mostly relying on deep learning [34,80]. Cloud then came up as an evolution of a series of technologies, mainly on virtualization and computer networks, which facilitated both workload management and interaction with remote resources respectively. Apart from software and hardware, cloud offers a business model where users pay for resources on demand. Compared to traditional HPC environments, in clouds users can quickly adjust their resource pools, via a mechanism known as elast...

show abstract

A hybrid HPC/cloud distributed infrastructure: Coupling EC2 cloud resources with HPC clusters to run large tightly coupled multiscale applications

Cited by 32 publications

References 24 publications

Multiscale computing in the exascale era

Multiscale computing in the exascale era

An efficient cloud scheduler design supporting preemptible instances

HPC Cloud for Scientific and Business Applications

Contact Info

Product

Resources

About