Science gateways - leveraging modeling and simulations in HPC infrastructures via increased usability

Gesing, Sandra; Dooley, Rion; Pierce, Marlon; Krüger, Jens; Grunzke, Richard; Herres‐Pawlis, Sonja; Hoffmann, Alexander

doi:10.1109/hpcsim.2015.7237017

“…For example, data repositories now support data analysis [71,72], science gateways facilitate the capture of rich provenance information [73], and publishers enable verification of figures and computational results from within papers [74], and through third party offerings [75,76].…”

Section: Related Workmentioning

confidence: 99%

Computing environments for reproducibility: Capturing the “Whole Tale”

Brinckman

¹

,

Chard

²

,

Gaffney³

et al. 2019

Future Generation Computer Systems

View full text Add to dashboard Cite

“…Science gateways are typically comprised of four layers: the user layer, application layer, job and data management layer, and the underlying computing infrastructures [59]. Gateway development tools, such as Apache Airavata [60], Agave [61], and Globus [62], facilitate the construction of science gateways by providing the necessary APIs and libraries.…”

Section: Cloud-based Scientific Servicesmentioning

confidence: 99%

“…Many science gateways offer domain-specific end-to-end research solutions, where data, analytical tools, and a compute infrastructure are delivered as a standalone service. Gateways are often tightly coupled with software stacks and the underlying compute infrastructure, thus increasing the usability of analytical applications and complex infrastructures [59]. Cloud computing provides an ideal platform for hosting scientific gateways [65,66,67].…”

Section: Cloud-based Scientific Servicesmentioning

confidence: 99%

Improving the Performance of Cloud-based Scientific Services

Chard¹

0

View full text Add to dashboard Cite

<p>Cloud computing provides access to a large scale set of readily available computing resources at the click of a button. The cloud paradigm has commoditised computing capacity and is often touted as a low-cost model for executing and scaling applications. However, there are significant technical challenges associated with selecting, acquiring, configuring, and managing cloud resources which can restrict the efficient utilisation of cloud capabilities. Scientific computing is increasingly hosted on cloud infrastructure—in which scientific capabilities are delivered to the broad scientific community via Internet-accessible services. This migration from on-premise to on-demand cloud infrastructure is motivated by the sporadic usage patterns of scientific workloads and the associated potential cost savings without the need to purchase, operate, and manage compute infrastructure—a task that few scientific users are trained to perform. However, cloud platforms are not an automatic solution. Their flexibility is derived from an enormous number of services and configuration options, which in turn result in significant complexity for the user. In fact, naïve cloud usage can result in poor performance and excessive costs, which are then directly passed on to researchers. This thesis presents methods for developing efficient cloud-based scientific services. Three real-world scientific services are analysed and a set of common requirements are derived. To address these requirements, this thesis explores automated and scalable methods for inferring network performance, considers various trade-offs (e.g., cost and performance) when provisioning instances, and profiles application performance, all in heterogeneous and dynamic cloud environments. Specifically, network tomography provides the mechanisms to infer network performance in dynamic and opaque cloud networks; cost-aware automated provisioning approaches enable services to consider, in real-time, various trade-offs such as cost, performance, and reliability; and automated application profiling allows a huge search space of applications, instance types, and configurations to be analysed to determine resource requirements and application performance. Finally, these contributions are integrated into an extensible and modular cloud provisioning and resource management service called SCRIMP. Cloud-based scientific applications and services can subscribe to SCRIMP to outsource their provisioning, usage, and management of cloud infrastructures. Collectively, the approaches presented in this thesis are shown to provide order of magnitude cost savings and significant performance improvement when employed by production scientific services.</p>

show abstract

“…Science gateways are typically comprised of four layers: the user layer, application layer, job and data management layer, and the underlying computing infrastructures [59]. Gateway development tools, such as Apache Airavata [60], Agave [61], and Globus [62], facilitate the construction of science gateways by providing the necessary APIs and libraries.…”

Section: Cloud-based Scientific Servicesmentioning

confidence: 99%

“…Many science gateways offer domain-specific end-to-end research solutions, where data, analytical tools, and a compute infrastructure are delivered as a standalone service. Gateways are often tightly coupled with software stacks and the underlying compute infrastructure, thus increasing the usability of analytical applications and complex infrastructures [59]. Cloud computing provides an ideal platform for hosting scientific gateways [65,66,67].…”

Section: Cloud-based Scientific Servicesmentioning

confidence: 99%

Improving the Performance of Cloud-based Scientific Services

Chard¹

0

View full text Add to dashboard Cite

<p>Cloud computing provides access to a large scale set of readily available computing resources at the click of a button. The cloud paradigm has commoditised computing capacity and is often touted as a low-cost model for executing and scaling applications. However, there are significant technical challenges associated with selecting, acquiring, configuring, and managing cloud resources which can restrict the efficient utilisation of cloud capabilities. Scientific computing is increasingly hosted on cloud infrastructure—in which scientific capabilities are delivered to the broad scientific community via Internet-accessible services. This migration from on-premise to on-demand cloud infrastructure is motivated by the sporadic usage patterns of scientific workloads and the associated potential cost savings without the need to purchase, operate, and manage compute infrastructure—a task that few scientific users are trained to perform. However, cloud platforms are not an automatic solution. Their flexibility is derived from an enormous number of services and configuration options, which in turn result in significant complexity for the user. In fact, naïve cloud usage can result in poor performance and excessive costs, which are then directly passed on to researchers. This thesis presents methods for developing efficient cloud-based scientific services. Three real-world scientific services are analysed and a set of common requirements are derived. To address these requirements, this thesis explores automated and scalable methods for inferring network performance, considers various trade-offs (e.g., cost and performance) when provisioning instances, and profiles application performance, all in heterogeneous and dynamic cloud environments. Specifically, network tomography provides the mechanisms to infer network performance in dynamic and opaque cloud networks; cost-aware automated provisioning approaches enable services to consider, in real-time, various trade-offs such as cost, performance, and reliability; and automated application profiling allows a huge search space of applications, instance types, and configurations to be analysed to determine resource requirements and application performance. Finally, these contributions are integrated into an extensible and modular cloud provisioning and resource management service called SCRIMP. Cloud-based scientific applications and services can subscribe to SCRIMP to outsource their provisioning, usage, and management of cloud infrastructures. Collectively, the approaches presented in this thesis are shown to provide order of magnitude cost savings and significant performance improvement when employed by production scientific services.</p>

show abstract

Science gateways - leveraging modeling and simulations in HPC infrastructures via increased usability

Cited by 3 publications

References 19 publications

Computing environments for reproducibility: Capturing the “Whole Tale”

Computing environments for reproducibility: Capturing the “Whole Tale”

Improving the Performance of Cloud-based Scientific Services

Improving the Performance of Cloud-based Scientific Services

Contact Info

Product

Resources

About