BeeFlow: A Workflow Management System for In Situ Processing across HPC and Cloud Systems

Chen, Jieyang; Guan, Qiang; Zhang, Zhao; Liang, Xin; Vernon, Louis J.; McPherson, Allen; Lo, Li-Ta; Grubel, Patricia; Randles, Tim; Ahrens, James

doi:10.1109/icdcs.2018.00103

Cited by 12 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We assume that the workflow description and generated DAG graph are provided to the scheduling system. One example of previous work done in BEEFlow [6], which proposed an in-situ analysis-enabled workflow management system that supports multiple platforms using HPC containers.…”

Section: A Mars System Overviewmentioning

confidence: 99%

MARS: Malleable Actor-Critic Reinforcement Learning Scheduler

Baheri

Tronge

Fang

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

Self Cite

View full text Add to dashboard Cite

Section: A Mars System Overviewmentioning

confidence: 99%

MARS: Malleable Actor-Critic Reinforcement Learning Scheduler

Baheri

Tronge

Fang

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

Self Cite

View full text Add to dashboard Cite

“…Researchers have been using "in-situ" to describe the situation when visualization/analysis programs can process simulation data without any data movements or if the visualization/analysis routines reside in the same processes with simulation [6,44]. There is also a broader definition of "in-situ" processing: processing data while it is generated [9,11,23,32]. More detailed terminology description and classification of in-situ has been introduced [10].…”

Section: In-situ Processing On Hpc Systemsmentioning

confidence: 99%

“…DagOn * [37] is a Python-based workflow engine that allows users to define parallel jobs represented by directed acyclic graphs on a combination of local machines, remote servers, and cloud-based infrastructures. Beeflow [9] is a workflow management system that supports traditional workflows as well as workflows with in situ analysis, and it utilizes events-synchronization primitives to enforce in situ workflow logic. Yildiz et al [49] explore the combination of task-based computing model and in situ workflows using Decaf and PyCOMPs, by integrating Decaf's in situ components as a sub-workflow of the task-based PyCOMPs workflow.…”

Section: Related Workmentioning

confidence: 99%

X-composer

Wang

Yan

et al. 2021

Proceedings of the Platform for Advanced Scientific Computing Conference

View full text Add to dashboard Cite

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

show abstract

“…These improvements are applicable and important for workflow systems, e.g., to help leverage both HPC and cloud resources for in situ processing [7], and to help envision unified resource management in the context of exascale computing [21].…”

Section: B Containerizationmentioning

confidence: 99%

Shared-Memory Communication for Containerized Workflows

Hobson

Yildiz

Nicolae

et al. 2021

2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

View full text Add to dashboard Cite

Scientific computation increasingly consists of a workflow of interrelated tasks. Containerization can make workflow systems more manageable, reproducible, and portable, but containers can impede communication due to their focus on encapsulation. In some circumstances, shared-memory regions are an effective way to improve performance of workflows; however sharing memory between containerized workflow tasks is difficult. In this work, we have created a software library called Dhmem that manages shared memory between workflow tasks in separate containers, with minimal code change and performance overhead. Instead of all code being in the same container, Dhmem allows a separate container for each workflow task to be constructed completely independently. Dhmem enables additional functionality: easy integration in existing workflow systems, communication configuration at runtime based on the environment, and scalable performance.

show abstract

BeeFlow: A Workflow Management System for In Situ Processing across HPC and Cloud Systems

Cited by 12 publications

References 16 publications

MARS: Malleable Actor-Critic Reinforcement Learning Scheduler

MARS: Malleable Actor-Critic Reinforcement Learning Scheduler

X-composer

Shared-Memory Communication for Containerized Workflows

Contact Info

Product

Resources

About