Experiences with distributed computing for meteorological applications: grid computing and cloud computing

Oesterle, Felix; Ostermann, Simon; Prodan, Radu; Mayr, Georg J.

doi:10.5194/gmd-8-2067-2015

Cited by 13 publications

(8 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The INDIGO IAM 8 (Identity and Access Management service) provides user identity and policy information to services so that consistent authorization decisions can be enforced across distributed services.…”

Section: Development Of Authorization and Authentication Infrastructuresmentioning

confidence: 99%

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

et al. 2018

View full text Add to dashboard Cite

This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.

show abstract

Section: Development Of Authorization and Authentication Infrastructuresmentioning

confidence: 99%

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Others have done such from other perspectives (Gupta and Milojicic 2011; Iosup et al 2011; Ismail and Khan 2015; Jackson et al 2010; Mauch 2015; Oesterle et al 2015; Sadooghi et al 2015; Thackston and Fortenberry 2015b; Yelick et al 2011; Zaspel and Griebel 2011). …”

Section: Running Neuron In the Cloudmentioning

confidence: 99%

Automating NEURON Simulation Deployment in Cloud Resources

Stockton

Santamarı́a

2016

Neuroinform

View full text Add to dashboard Cite

Simulations in neuroscience are performed on local servers or High Performance Computing (HPC) facilities. Recently, cloud computing has emerged as a potential computational platform for neuroscience simulation. In this paper we compare and contrast HPC and cloud resources for scientific computation, then report how we deployed NEURON, a widely used simulator of neuronal activity, in three clouds: Chameleon Cloud, a hybrid private academic cloud for cloud technology research based on the Open-Stack software; Rackspace, a public commercial cloud, also based on OpenStack; and Amazon Elastic Cloud Computing, based on Amazon’s proprietary software. We describe the manual procedures and how to automate cloud operations. We describe extending our simulation automation software called NeuroManager (Stockton and Santamaria, Frontiers in Neuroinformatics, 2015), so that the user is capable of recruiting private cloud, public cloud, HPC, and local servers simultaneously with a simple common interface. We conclude by performing several studies in which we examine speedup, efficiency, total session time, and cost for sets of simulations of a published NEURON model.

show abstract

“…Cloud platforms are found to be efficient for small-tomedium sized simulations with less than 100 CPU cores, but the typically slower inter-node communication on the cloud can affect the parallel efficiency of larger simulations. Cost comparisons between cloud platforms and traditional clusters show inconsistent results, either in favor of the cloud (Roloff et al 2012;Huang et al 2013;Oesterle et al 2015;Thackston and Fortenberry 2015;Dodson et al 2016) or local clusters (Carlyle et al 2010;Freniere et al 2016;Emeras et al 2017;Chang et al 2018), depending on assumptions regarding resource utilization, parallelization efficiency, storage requirement, and billing model. The cloud is particularly cost-effective for occasional or intermittent workloads.…”

Section: Introductionmentioning

confidence: 99%

Enabling Immediate Access to Earth Science Models through Cloud Computing: Application to the GEOS-Chem Model

Zhuang

Jacob

Gaya

et al. 2019

Bulletin of the American Meteorological Society

View full text Add to dashboard Cite

Cloud computing platforms can provide fast and easy access to complex Earth science models and large datasets. This article presents a mature capability for running the GEOS-Chem global 3D model of atmospheric chemistry on the Amazon Web Services (AWS) cloud. GEOS-Chem users at any experience level can get immediate access to the latest, standard version of the model in a preconfigured software environment with all needed meteorological and other input data, and they can analyze model output data easily within the cloud using Python tools in Jupyter notebooks. Users with no prior knowledge of cloud computing are provided with easy-to-follow, step-by-step instructions. They can learn how to complete a demo project in less than one hour, and from there they can configure and submit their own simulations. The cloud is particularly attractive for beginning and occasional users who otherwise may need to spend substantial time configuring a local computing environment. Heavy users with their own local clusters can also benefit from the cloud to access the latest standard model and datasets, share simulation configurations and results, benchmark local simulations, and respond to surges in computing demand. Software containers allow GEOS-Chem and its software environment to be moved smoothly between cloud platforms and local clusters, so that the exact same simulation can be reproduced everywhere. Because the software requirements and workflows tend to be similar across Earth science models, the work presented here provides general guidance for porting models to cloud computing platforms in a user-accessible way.

show abstract

Experiences with distributed computing for meteorological applications: grid computing and cloud computing

Cited by 13 publications

References 29 publications

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

Automating NEURON Simulation Deployment in Cloud Resources

Enabling Immediate Access to Earth Science Models through Cloud Computing: Application to the GEOS-Chem Model

Contact Info

Product

Resources

About