A Comprehensive Perspective on Pilot-Job Systems

Turilli, Matteo; Santcroos, Mark; Jha, Shantenu

doi:10.1145/3177851

Cited by 69 publications

(70 citation statements)

References 156 publications

(145 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ref. [17] provided the architectural paradigm for pilot systems, however it is still unclear how an analogous paradigm would complement the work done on reference architectures for workflow systems [10], [8], and whether, given the very broad diversity of workflow systems and tools, we can even formulate a single architectural paradigm. This paradigm has been elusive so far, but it might be more fruitful to formulate system-level paradigms that have the properties of building blocks.…”

Section: Discussionmentioning

confidence: 99%

Middleware Building Blocks for Workflow Systems

et al. 2019

Self Cite

View full text Add to dashboard Cite

This paper describes a building blocks approach to the design of scientific workflow systems. We discuss RADICAL-Cybertools as one implementation of the building blocks concept, showing how they are designed and developed in accordance with this approach. This paper offers three main contributions: (i) showing the relevance of the design principles underlying the building blocks approach to support scientific workflows on high performance computing platforms; (ii) illustrating a set of building blocks that enable multiple points of integration, "unifying" conceptual reasoning across otherwise very different tools and systems; and (iii) case studies discussing how RADICAL-Cybertools are integrated with existing workflow, workload, and general purpose computing systems and used to develop domainspecific workflow systems.

show abstract

Section: Discussionmentioning

confidence: 99%

Middleware Building Blocks for Workflow Systems

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…EnTK provides the ability to create and execute ensemble-based workflows/applications with diverse coordination and communication algorithms, abstracting the need for explicit resource management. EnTK uses RP as a pilot-based [28] runtime system to provide resource management and task execution capabilities. In turn, RP uses RS as an access layer towards HPC resources.…”

Section: A Radical-cybertools: Ensemble Execution On Summitmentioning

confidence: 99%

DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding

Lee

Turilli

Jha

et al. 2019

2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS)

Self Cite

View full text Add to dashboard Cite

Simulations of biological macromolecules play an important role in understanding the physical basis of a number of complex processes such as protein folding. Even with increasing computational power and evolution of specialized architectures, the ability to simulate protein folding at atomistic scales still remains challenging. This stems from the dual aspects of high dimensionality of protein conformational landscapes, and the inability of atomistic molecular dynamics (MD) simulations to sufficiently sample these landscapes to observe folding events. Machine learning/deep learning (ML/DL) techniques, when combined with atomistic MD simulations offer the opportunity to potentially overcome these limitations by: (1) effectively reducing the dimensionality of MD simulations to automatically build latent representations that correspond to biophysically relevant reaction coordinates (RCs), and (2) driving MD simulations to automatically sample potentially novel conformational states based on these RCs. We examine how coupling DL approaches with MD simulations can fold small proteins effectively on supercomputers. In particular, we study the computational costs and effectiveness of scaling DL-coupled MD workflows by folding two prototypical systems, viz., Fs-peptide and the fast-folding variant of the villin head piece protein. We demonstrate that a DL driven MD workflow is able to effectively learn latent representations and drive adaptive simulations. Compared to traditional MD-based approaches, our approach achieves an effective performance gain in sampling the folded states by at least 2.3x. Our study provides a quantitative basis to understand how DL driven MD simulations, can lead to effective performance gains and reduced times to solution on supercomputing resources.

show abstract

“…RADICAL-Pilot [25] is a Pilot system that implements the pilot paradigm as outlined in Ref. [39]. RADICAL-Pilot (RP) is implemented in Python and provides a well defined API and usage modes.…”

Section: Radical-pilotmentioning

confidence: 99%

Task-parallel Analysis of Molecular Dynamics Trajectories

Paraskevakos

Luckow

Khoshlessan

et al. 2018

Proceedings of the 47th International Conference on Parallel Processing

Self Cite

View full text Add to dashboard Cite

Different parallel frameworks for implementing data analysis applications have been proposed by the HPC and Big Data communities. In this paper, we investigate three task-parallel frameworks: Spark, Dask and RADICAL-Pilot with respect to their ability to support data analytics on HPC resources and compare them to MPI. We investigate the data analysis requirements of Molecular Dynamics (MD) simulations which are significant consumers of supercomputing cycles, producing immense amounts of data. A typical large-scale MD simulation of a physical system of O(100k) atoms over µsecs can produce from O(10) GB to O(1000) GBs of data. We propose and evaluate different approaches for parallelization of a representative set of MD trajectory analysis algorithms, in particular the computation of path similarity and leaflet identification. We evaluate Spark, Dask and RADICAL-Pilot with respect to their abstractions and runtime engine capabilities to support these algorithms. We provide a conceptual basis for comparing and understanding different frameworks that enable users to select the optimal system for each application. We also provide a quantitative performance analysis of the different algorithms across the three frameworks.

show abstract

A Comprehensive Perspective on Pilot-Job Systems

Cited by 69 publications

References 156 publications

Middleware Building Blocks for Workflow Systems

Middleware Building Blocks for Workflow Systems

DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding

Task-parallel Analysis of Molecular Dynamics Trajectories

Contact Info

Product

Resources

About