SAGA: A standardized access layer to heterogeneous Distributed Computing Infrastructure

Merzky, André; Weidner, Ole; Jha, Shantenu

doi:10.1016/j.softx.2015.03.001

Cited by 39 publications

(31 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RCT consists of three main components: RADICAL-Ensemble Toolkit (EnTK) [23], [24], RADICAL-Pilot (RP) [25], and RADICAL-SAGA(RS) [26], [27]. EnTK provides the ability to create and execute ensemble-based workflows/applications with diverse coordination and communication algorithms, abstracting the need for explicit resource management.…”

Section: A Radical-cybertools: Ensemble Execution On Summitmentioning

confidence: 99%

DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding

Lee

Turilli

Jha

et al. 2019

2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS)

Self Cite

View full text Add to dashboard Cite

Simulations of biological macromolecules play an important role in understanding the physical basis of a number of complex processes such as protein folding. Even with increasing computational power and evolution of specialized architectures, the ability to simulate protein folding at atomistic scales still remains challenging. This stems from the dual aspects of high dimensionality of protein conformational landscapes, and the inability of atomistic molecular dynamics (MD) simulations to sufficiently sample these landscapes to observe folding events. Machine learning/deep learning (ML/DL) techniques, when combined with atomistic MD simulations offer the opportunity to potentially overcome these limitations by: (1) effectively reducing the dimensionality of MD simulations to automatically build latent representations that correspond to biophysically relevant reaction coordinates (RCs), and (2) driving MD simulations to automatically sample potentially novel conformational states based on these RCs. We examine how coupling DL approaches with MD simulations can fold small proteins effectively on supercomputers. In particular, we study the computational costs and effectiveness of scaling DL-coupled MD workflows by folding two prototypical systems, viz., Fs-peptide and the fast-folding variant of the villin head piece protein. We demonstrate that a DL driven MD workflow is able to effectively learn latent representations and drive adaptive simulations. Compared to traditional MD-based approaches, our approach achieves an effective performance gain in sampling the folded states by at least 2.3x. Our study provides a quantitative basis to understand how DL driven MD simulations, can lead to effective performance gains and reduced times to solution on supercomputing resources.

show abstract

Section: A Radical-cybertools: Ensemble Execution On Summitmentioning

confidence: 99%

DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding

Lee

Turilli

Jha

et al. 2019

2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Figure 2 illustrates the architecture of SAGA-Hadoop. SAGA-Hadoop uses SAGA [35] to spawn and control Hadoop clusters inside an environment managed by an HPC scheduler, such as PBS, SLURM or SGE, or clouds. SAGA is used for dispatching a bootstrap process that generates the necessary configuration files and for starting the Hadoop processes.…”

Section: Integrating Hadoop and Spark With Radical-pilotmentioning

confidence: 99%

Hadoop on HPC: Integrating Hadoop and Pilot-Based Dynamic Resource Management

Luckow

Paraskevakos

Chantzialexiou

et al. 2016

2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

Self Cite

View full text Add to dashboard Cite

Abstract-High-performance computing platforms such as "supercomputers" have traditionally been designed to meet the compute demands of scientific applications. Consequently, they have been architected as net producers and not consumers of data. The Apache Hadoop ecosystem has evolved to meet the requirements of data processing applications and has addressed many of the traditional limitations of HPC platforms. There exist a class of scientific applications however, that need the collective capabilities of traditional high-performance computing environments and the Apache Hadoop ecosystem. For example, the scientific domains of bio-molecular dynamics, genomics and network science need to couple traditional computing with Hadoop/Spark based analysis. We investigate the critical question of how to present the capabilities of both computing environments to such scientific applications. Whereas this questions needs answers at multiple levels, we focus on the design of resource management middleware that might support the needs of both. We propose extensions to the Pilot-Abstraction so as to provide a unifying resource management layer. This is an important step towards interoperable use of HPC and Hadoop/Spark. It also allows applications to integrate HPC stages (e. g. simulations) to data analytics. Many supercomputing centers have started to officially support Hadoop environments, either in a dedicated environment or in hybrid deployments using tools such as myHadoop. This typically involves many intrinsic, environment-specific details that need to be mastered, and often swamp conceptual issues like: How best to couple HPC and Hadoop application stages? How to explore runtime trade-offs (data localities vs. data movement)? This paper provides both conceptual understanding and practical solutions to the integrated use of HPC and Hadoop environments. Our experiments are performed on state-of-the-art production HPC environments and provide middleware for multiple domain sciences.

show abstract

“…Compared to other open source GIS projects like SAGA (Merzky et al, 2015) or QGIS (2015), SAMT2 is a small tool kit. SAMT2 is specialized to fill the gap between matrix calculation tools such as Matlab or the open source clone Octave (Alsberg and Hagen, 2006) and a GIS.…”

Section: Accepted Manuscriptmentioning

confidence: 99%

Spatial Analysis and Modeling Tool Version 2 (SAMT2), a spatial modeling tool kit written in Python

Wieland

Groth

Linde

et al. 2015

Ecological Informatics

View full text Add to dashboard Cite

SAGA: A standardized access layer to heterogeneous Distributed Computing Infrastructure

Cited by 39 publications

References 15 publications

DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding

DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding

Hadoop on HPC: Integrating Hadoop and Pilot-Based Dynamic Resource Management

Spatial Analysis and Modeling Tool Version 2 (SAMT2), a spatial modeling tool kit written in Python

Contact Info

Product

Resources

About