CWL-Airflow: a lightweight pipeline manager supporting Common Workflow Language

Kotliar, Michael; Kartashov, Andrey V.; Barski, Artem

doi:10.1093/gigascience/giz084

Cited by 46 publications

(24 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…They were typically built as a sequential series of commands that needed to be reproduced for every sample. Nowadays many bioinformaticians [ 5 – 7 ] rely on workflow management systems (WMSs) [ 8 – 11 ] to improve execution efficiency and scalability. WMSs usually accept different programming languages within the same pipeline and can be run by a single command.…”

Section: Introductionmentioning

confidence: 99%

Impact of concurrency on the performance of a whole exome sequencing pipeline

et al. 2021

View full text Add to dashboard Cite

Background Current high-throughput technologies—i.e. whole genome sequencing, RNA-Seq, ChIP-Seq, etc.—generate huge amounts of data and their usage gets more widespread with each passing year. Complex analysis pipelines involving several computationally-intensive steps have to be applied on an increasing number of samples. Workflow management systems allow parallelization and a more efficient usage of computational power. Nevertheless, this mostly happens by assigning the available cores to a single or few samples’ pipeline at a time. We refer to this approach as naive parallel strategy (NPS). Here, we discuss an alternative approach, which we refer to as concurrent execution strategy (CES), which equally distributes the available processors across every sample’s pipeline. Results Theoretically, we show that the CES results, under loose conditions, in a substantial speedup, with an ideal gain range spanning from 1 to the number of samples. Also, we observe that the CES yields even faster executions since parallelly computable tasks scale sub-linearly. Practically, we tested both strategies on a whole exome sequencing pipeline applied to three publicly available matched tumour-normal sample pairs of gastrointestinal stromal tumour. The CES achieved speedups in latency up to 2–2.4 compared to the NPS. Conclusions Our results hint that if resources distribution is further tailored to fit specific situations, an even greater gain in performance of multiple samples pipelines execution could be achieved. For this to be feasible, a benchmarking of the tools included in the pipeline would be necessary. It is our opinion these benchmarks should be consistently performed by the tools’ developers. Finally, these results suggest that concurrent strategies might also lead to energy and cost savings by making feasible the usage of low power machine clusters.

show abstract

Section: Introductionmentioning

confidence: 99%

Impact of concurrency on the performance of a whole exome sequencing pipeline

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Open source CWL Pipelines used here are available at https://github.com/datirium/workflows . As a workflow runner we used CWL-Airflow ( Kotliar et al., 2019 ), however, the same pipelines ( https://doi.org/10.5281/zenodo.5339177 ) can be executed in any other CWL-based execution environments ( https://commonwl.org ).…”

Section: Step-by-step Methods Detailsmentioning

confidence: 99%

Preparation of mouse pancreatic tumor for single-cell RNA sequencing and analysis of the data

et al. 2021

Self Cite

View full text Add to dashboard Cite

Summary Preparation of single-cell suspension from primary tumor tissue can provide a valuable resource for functional, genetic, proteomic, and tumor microenvironment studies. Here, we describe an effective protocol for mouse pancreatic tumor dissociation with further processing of tumor suspension for single-cell RNA sequencing analysis of cellular populations. We further provide an outline of the bioinformatics processing of the data and clustering of heterogeneous cellular populations comprising pancreatic tumors using Common Workflow Language (CWL) pipelines within user-friendly Scientific Data Analysis Platform ( https://SciDAP.com ). For complete details on the use and execution of this protocol, please refer to Gabitova-Cornell et al. (2020) .

show abstract

“…Our workflow script code under the JSON format applied in this paper is publicly accessible (Findable and Accessible) and can be implemented on the CGC platform (Interoperable and Reusable). Our code can also be executed using other CWL executors, such as CWL-Tool and CWL-Airflow [41]. However, because the code contained a platform-specific field, it might not work outside the CGC platform.…”

Section: Limitationsmentioning

confidence: 99%

Orchestrating an Optimized Next-Generation Sequencing-Based Cloud Workflow for Robust Viral Identification during Pandemics

Lim

Hsiao

Lee

2021

Biology

View full text Add to dashboard Cite

Coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has recently become a novel pandemic event following the swine flu that occurred in 2009, which was caused by the influenza A virus (H1N1 subtype). The accurate identification of the huge number of samples during a pandemic still remains a challenge. In this study, we integrate two technologies, next-generation sequencing and cloud computing, into an optimized workflow version that uses a specific identification algorithm on the designated cloud platform. We use 182 samples (92 for COVID-19 and 90 for swine flu) with short-read sequencing data from two open-access datasets to represent each pandemic and evaluate our workflow performance based on an index specifically created for SARS-CoV-2 or H1N1. Results show that our workflow could differentiate cases between the two pandemics with a higher accuracy depending on the index used, especially when the index that exclusively represented each dataset was used. Our workflow substantially outperforms the original complete identification workflow available on the same platform in terms of time and cost by preserving essential tools internally. Our workflow can serve as a powerful tool for the robust identification of cases and, thus, aid in controlling the current and future pandemics.

show abstract

CWL-Airflow: a lightweight pipeline manager supporting Common Workflow Language

Cited by 46 publications

References 18 publications

Impact of concurrency on the performance of a whole exome sequencing pipeline

Impact of concurrency on the performance of a whole exome sequencing pipeline

Preparation of mouse pancreatic tumor for single-cell RNA sequencing and analysis of the data

Orchestrating an Optimized Next-Generation Sequencing-Based Cloud Workflow for Robust Viral Identification during Pandemics

Contact Info

Product

Resources

About