A Lightweight Model for Right-Sizing Master-Worker Applications

Kremer-Herman, Nathaniel; Tovar, Benjamín; Thain, Douglas

doi:10.1109/sc.2018.00042

Cited by 8 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multiple constrained optimizations that are specified at the same step in the workflow can be carried out in parallel. In standalone operation mode, TorsionDrive uses the Work Queue distributed computing framework 35,36 to take advantage of parallel resources. When TorsionDrive is used as part of the QCArchive ecosystem, 33 it works as an API to specify the constrained optimization inputs while QCArchive is responsible for job management; this is described in detail in Section 4.…”

Section: E(φ Abcdmentioning

confidence: 99%

“…Task execution systems not only allow the TorsionDrive calculation to be parallelized across cores of a single node but across computational nodes even if the underlying quantum chemistry program is not able to do so. There are many such task execution systems available such as Work Queue, 35,36 Dask, 51 Parsl, 52 RADICAL Pilot, 53…”

Section: Task Execution Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Driving torsion scans with wavefront propagation

Qiu

Smith

Stern

et al. 2020

The Journal of Chemical Physics

View full text Add to dashboard Cite

The parameterization of torsional / dihedral angle potential energy terms is a crucial part of developing molecular mechanics force fields.Quantum mechanical (QM) methods are often used to provide samples of the potential energy surface (PES) for fitting the empirical parameters in these force field terms.To ensure that the sampled molecular configurations are thermodynamically feasible, constrained QM geometry optimizations are typically carried out, which relax the orthogonal degrees of freedom while fixing the target torsion angle(s) on a grid of values.However, the quality of results and computational cost are affected by various factors on a non-trivial PES, such as dependence on the chosen scan direction and the lack of efficient approaches to integrate results started from multiple initial guesses.In this paper we propose a systematic and versatile workflow called \textit{TorsionDrive} to generate energy-minimized structures on a grid of torsion constraints by means of a recursive wavefront propagation algorithm, which resolves the deficiencies of conventional scanning approaches and generates higher quality QM data for force field development.The capabilities of our method are presented for multi-dimensional scans and multiple initial guess structures, and an integration with the MolSSI QCArchive distributed computing ecosystem is described.The method is implemented in an open-source software package that is compatible with many QM software packages and energy minimization codes. File list (2)download file view on ChemRxiv TorsionDrive_manuscript_submitted.pdf (3.57 MiB) download file view on ChemRxiv TorsionDrive_SI.zip (5.32 MiB)

show abstract

Section: E(φ Abcdmentioning

confidence: 99%

Section: Task Execution Systemsmentioning

confidence: 99%

Driving torsion scans with wavefront propagation

Qiu

Smith

Stern

et al. 2020

The Journal of Chemical Physics

View full text Add to dashboard Cite

show abstract

“…Increasingly statistical methods are used to understand performance and to make predictions, e. g., for resource (re)configurations decisions [22]. Kremer-Herman et al [23] propose a model for recommending the optimal infrastructure configuration for master/worker applications. Ernest [24] is a system that combines analytical modeling and data derived from a small set of performance counters and sample runs.…”

Section: B Streaming Performance and Modelingmentioning

confidence: 99%

Performance Characterization and Modeling of Serverless and HPC Streaming Applications

Luckow

Jha

2019

2019 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Experiment-in-the-Loop Computing (EILC) requires support for numerous types of processing and the management of heterogeneous infrastructure over a dynamic range of scales: from the edge to the cloud and HPC, and intermediate resources. Serverless is an emerging service that combines high-level middleware services, such as distributed execution engines for managing tasks, with low-level infrastructure. It offers the potential of usability and scalability, but adds to the complexity of managing heterogeneous and dynamic resources. In response, we extend Pilot-Streaming to support serverless platforms. Pilot-Streaming provides a unified abstraction for resource management for HPC, cloud, and serverless, and allocates resource containers independent of the application workload removing the need to write resource-specific code. Understanding of the performance and scaling characteristics of streaming applications and infrastructure presents another challenge for EILC. StreamInsight provides insight into the performance of streaming applications and infrastructure, their selection, configuration and scaling behavior. Underlying StreamInsight is the universal scalability law, which permits the accurate quantification of scalability properties of streaming applications. Using experiments on HPC and AWS Lambda, we demonstrate that StreamInsight provides an accurate model for a variety of application characteristics, e. g., machine learning model sizes and resource configurations.

show abstract

“…Tasks are shipped from the central scheduler to a worker process, the results of the task are shipped back to the central task scheduler.Task execution systems not only allow the TorsionDrive calculation to be parallelized across cores of a single node but across computational nodes even if the underlying quantum chemistry program is not able to do so. There are many such task execution systems available such as Work Queue,35,36 Dask,51 Parsl,52 RADICAL Pilot,53 and Fireworks 54 in the academic computing space which provide this service and can be trivially integrated with TorsionDrive. At present, TorsionDrive supports Work Queue when running in standalone operation mode, and through the QCArchive integration supports several other task execution systems such as Dask, Parsl, RADICAL, and Fireworks.…”

mentioning

confidence: 99%

Driving Torsion Scans with Wavefront Propagation

Qiu¹,

Smith²,

Stern³

et al. 2020

Preprint

View full text Add to dashboard Cite

<div>The parameterization of torsional / dihedral angle potential energy terms is a crucial part of developing molecular mechanics force fields.</div><div>Quantum mechanical (QM) methods are often used to provide samples of the potential energy surface (PES) for fitting the empirical parameters in these force field terms.</div><div>To ensure that the sampled molecular configurations are thermodynamically feasible, constrained QM geometry optimizations are typically carried out, which relax the orthogonal degrees of freedom while fixing the target torsion angle(s) on a grid of values.</div><div>However, the quality of results and computational cost are affected by various factors on a non-trivial PES, such as dependence on the chosen scan direction and the lack of efficient approaches to integrate results started from multiple initial guesses.</div><div>In this paper we propose a systematic and versatile workflow called \textit{TorsionDrive} to generate energy-minimized structures on a grid of torsion constraints by means of a recursive wavefront propagation algorithm, which resolves the deficiencies of conventional scanning approaches and generates higher quality QM data for force field development.</div><div>The capabilities of our method are presented for multi-dimensional scans and multiple initial guess structures, and an integration with the MolSSI QCArchive distributed computing ecosystem is described.</div><div>The method is implemented in an open-source software package that is compatible with many QM software packages and energy minimization codes.</div>

show abstract

A Lightweight Model for Right-Sizing Master-Worker Applications

Cited by 8 publications

References 24 publications

Driving torsion scans with wavefront propagation

Driving torsion scans with wavefront propagation

Performance Characterization and Modeling of Serverless and HPC Streaming Applications

Driving Torsion Scans with Wavefront Propagation

Contact Info

Product

Resources

About