WorkflowHub: Community Framework for Enabling Scientific Workflow Research and Development

Silva, Rafael Ferreira da; Pottier, Loïc; Coleman, Tainã; Deelman, Ewa; Casanova, Henri

doi:10.1109/works51914.2020.00012

Cited by 16 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Once accepted, each workflow is continuously checked for best-practice conformance and tested during each new Galaxy release. The workflows are also automatically published to Dockstore ( 12 ) and WorkflowHub ( 13 ) and optionally synchronized to a list of Galaxy servers.…”

Section: Scientific Applications and Usesmentioning

confidence: 99%

The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update

Ostrovsky¹,

Mahmoud²,

Lonie³

et al. 2022

Nucleic Acids Research

672

232

View full text Add to dashboard Cite

Galaxy is a mature, browser accessible workbench for scientific computing. It enables scientists to share, analyze and visualize their own data, with minimal technical impediments. A thriving global community continues to use, maintain and contribute to the project, with support from multiple national infrastructure providers that enable freely accessible analysis and training services. The Galaxy Training Network supports free, self-directed, virtual training with >230 integrated tutorials. Project engagement metrics have continued to grow over the last 2 years, including source code contributions, publications, software packages wrapped as tools, registered users and their daily analysis jobs, and new independent specialized servers. Key Galaxy technical developments include an improved user interface for launching large-scale analyses with many files, interactive tools for exploratory data analysis, and a complete suite of machine learning tools. Important scientific developments enabled by Galaxy include Vertebrate Genome Project (VGP) assembly workflows and global SARS-CoV-2 collaborations.

show abstract

Section: Scientific Applications and Usesmentioning

confidence: 99%

The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update

Ostrovsky¹,

Mahmoud²,

Lonie³

et al. 2022

Nucleic Acids Research

672

232

View full text Add to dashboard Cite

show abstract

“…Providing machine‐readable APIs for HPC resources is the logical prerequisite to make this connection happen. It is also particularly fitting these days as the workflows community comes together to discuss common needs which, in turn, can inform the development of such APIs 14 …”

Section: Superfacility Apimentioning

confidence: 99%

Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis

Blaschke,

Brewster,

Paley

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryX‐ray scattering experiments using free electron lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID‐19 viral proteins). XFEL experiments are a challenge to computing in two ways: (i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; (ii) data‐collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min—sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS‐NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Completing data analysis within 10 min is a first for XFEL experiments and an important milestone if we are to keep up with data‐collection trends.

show abstract

“…Scientific workflows (SWF) are a cornerstone of modern scientific computing. They are used for complex computational applications that require robust management for big data that are typically stored and processed at heterogeneous, distributed resources [30]. SWF systems make scientists focus on their research rather than on details of computation management.…”

Section: The Computational Workflowmentioning

confidence: 99%

Micro-Workflows Data Stream Processing Model for Industrial Internet of Things

2021

JSFI

View full text Add to dashboard Cite

The fog computing paradigm has become prominent in stream processing for IoT systems where cloud computing struggles from high latency challenges. It enables the deployment of computational resources between the edge and cloud layers and helps to resolve constraints, primarily due to the need to react in real-time to state changes, improve the locality of data storage, and overcome external communication channels' limitations. There is an urgent need for tools and platforms to model, implement, manage, and monitor complex fog computing workflows. Traditional scientific workflow management systems (SWMSs) provide modularity and flexibility to design, execute, and monitor complex computational workflows used in smart industry applications. However, they are mainly focused on batch execution of jobs consisting of tightly coupled tasks. Integrating data streams into SWMSs of IoT systems is challenging. We proposed a microworkflow model to redesign the monolith architecture of workflow systems into a set of smaller and independent workflows that support stream processing. Micro-workflow is an independent data stream processing service that can be deployed on different layers of the fog computing environment. To validate the feasibility and practicability of the micro-workflow refactoring, we provide intensive experimental analysis evaluating the interval between sensor messages, the time interval required to create a message, between sending sensor message and receiving the message in SWMS, including data serialization, network latency, etc. We show that the proposed decoupling support of the independence of implementation, execution, development, maintenance, and cross-platform deployment, where each micro-workflow becomes a standalone computational unit, is a suitable mechanism for IoT stream processing.

show abstract

WorkflowHub: Community Framework for Enabling Scientific Workflow Research and Development

Cited by 16 publications

References 19 publications

The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update

The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update

Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis

Micro-Workflows Data Stream Processing Model for Industrial Internet of Things

Contact Info

Product

Resources

About