Synapse: Synthetic application profiler and emulator

Merzky, André; Ha, Ming Tai; Turilli, Matteo; Jha, Shantenu

doi:10.1016/j.jocs.2018.06.012

Cited by 9 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To account for that, often simplified, synthetic workloads are used to study performance (e. g., the Mini-App framework [32]). Similar techniques are commonly used for generating reproducible data and compute workloads, see [33], [34], [35].…”

Section: Discussionmentioning

confidence: 99%

Methods and Experiences for Developing Abstractions for Data-intensive, Scientific Applications

Luckow¹,

Jha²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Developing software for scientific applications that require the integration of diverse types of computing, instruments, and data present challenges that are distinct from commercial software. These applications require scale, and the need to integrate various programming and computational models with evolving and heterogeneous infrastructure. Pervasive and effective abstractions for distributed infrastructures are thus critical; however, the process of developing abstractions for scientific applications and infrastructures is not well understood. While theory-based approaches for system development are suited for well-defined, closed environments, they have severe limitations for designing abstractions for scientific systems and applications. The design science research (DSR) method provides the basis for designing practical systems that can handle real-world complexities at all levels. In contrast to theory-centric approaches, DSR emphasizes both practical relevance and knowledge creation by building and rigorously evaluating all artifacts. We show how DSR provides a well-defined framework for developing abstractions and middleware systems for distributed systems. Specifically, we address the critical problem of distributed resource management on heterogeneous infrastructure over a dynamic range of scales, a challenge that currently limits many scientific applications. We use the pilot-abstraction, a widely used resource management abstraction for high-performance, high throughput, big data, and streaming applications, as a case study for evaluating the DSR activities. For this purpose, we analyze the research process and artifacts produced during the design and evaluation of the pilot-abstraction. We find DSR provides a concise framework for iteratively designing and evaluating systems. Finally, we capture our experiences and formulate different lessons learned.

show abstract

Section: Discussionmentioning

confidence: 99%

Methods and Experiences for Developing Abstractions for Data-intensive, Scientific Applications

Luckow¹,

Jha²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Additionally, tools are available to generate synthetic workloads. Both generic tools [35] and neuroscience application specific tools exist [22,36]. The combination of these two sets of tools will allow validation of functionality, even in the absence of fully implemented use cases.…”

Section: Discussionmentioning

confidence: 99%

Staged deployment of interactive multi-application HPC workflows

Klijn

Diaz-Pier

Morrison

et al. 2019

2019 International Conference on High Performance Computing &Amp; Simulation (HPCS)

View full text Add to dashboard Cite

Running scientific workflows on a supercomputer can be a daunting task for a scientific domain specialist. Workflow management solutions (WMS) are a standard method for reducing the complexity of application deployment on high performance computing (HPC) infrastructure. We introduce the design for a middleware system that extends and combines the functionality from existing solutions in order to create a high-level, staged usercentric operation/deployment model. This design addresses the requirements of several use cases in the life sciences, with a focus on neuroscience. In this manuscript we focus on two use cases: 1) three coupled neuronal simulators (for three different space/time scales) with in-transit visualization and 2) a closed-loop workflow optimized by machine learning, coupling a robot with a neural network simulation. We provide a detailed overview of the application-integrated monitoring in relationship with the HPC job. We present here a novel usage model for large scale interactive multi-application workflows running on HPC systems which aims at reducing the complexity of deployment and execution, thus enabling new science.

show abstract

“…Such fluctuations would make the separation of RP overheads from resource fluctuations and runtime variations of the application's tasks difficult, if not impossible. Thus, we profiled and emulated GROMACS simulations with Synapse [44]. Synapse profiles the compute, Synapse offers our experiments several advantages over the direct use of the executable it emulates: (1) simplified and selfcontained deployment without third parties libraries and compilers dependences; (2) high-fidelity replication of the computing patterns of the emulated executable without actual input/output files; (3) profiling capabilities independent of third parties applications; (4) control over the number of FLOPs executed; and (5) selective emulation of the type of profiled resources.…”

Section: A Experiments Designmentioning

confidence: 99%

Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

Merzky¹,

Turilli²,

Титов³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Many extreme scale scientific applications have workloads comprised of a large number of individual highperformance tasks. The Pilot abstraction decouples workload specification, resource management, and task execution via job placeholders and late-binding. As such, suitable implementations of the Pilot abstraction can support the collective execution of large number of tasks on supercomputers. We introduce RADICAL-Pilot (RP) as a portable, modular and extensible Pilot enabled runtime system. We describe RP's design, architecture and implementation. We characterize its performance and show its ability to scalably execute workloads comprised of tens of thousands heterogeneous tasks on DOE and NSF leadership-class HPC platforms. Specifically, we investigate RP's weak/strong scaling with CPU/GPU, single/multi core, (non)MPI tasks and python functions when using most of ORNL Summit and TACC Frontera. RADICAL-Pilot can be used stand-alone, as well as the runtime for third-party workflow systems.

show abstract

Synapse: Synthetic application profiler and emulator

Cited by 9 publications

References 15 publications

Methods and Experiences for Developing Abstractions for Data-intensive, Scientific Applications

Methods and Experiences for Developing Abstractions for Data-intensive, Scientific Applications

Staged deployment of interactive multi-application HPC workflows

Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

Contact Info

Product

Resources

About