Making Canonical Workflow Building Blocks Interoperable across Workflow Languages

Soiland‐Reyes, Stian; Bayarri, Genís; Andrio, Pau; Long, R. E.; Lowe, Douglas; Niewielska, Ania; Hospital, Adam; Groth, Paul

doi:10.1162/dint_a_00135

Cited by 4 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, the presented workflows are transversal, common, and useful pipelines that can be applied as sub-workflows in more complex studies or different platforms. In fact, these demonstration workflows were converted, when possible, to different formats, thanks to the compatibility of the BioBB library with a range of workflow managers [ 19 ]. Common Workflow Language (CWL) [ 48 ], Galaxy [ 49 ], and pure Python versions of the presented workflows are available from a centralized GitHub repository ( https://github.com/bioexcel/biobb_workflows/ ).…”

Section: Discussion/conclusionmentioning

confidence: 99%

“…The digital notebooks presented in this work are strongly reliant on Jupyter Notebooks [ 54 ] and Python programming language. The BioBB library [ 17 , 19 ] is the main engine behind the workflows. BioPython library [ 55 ] is heavily used in the BioBB modules to process and work with PDB files.…”

Section: Methodsmentioning

confidence: 99%

“…Designed following the recommendations of the ELIXIR European Life Science infrastructure and the FAIR principles for software development, the library provides a set of interoperable units based on a collection of Python wrappers encapsulating software components. It is freely accessible from GitHub and BioConda packages [ 18 ], installable via Conda, Docker and Singularity containers, and compatible with several workflow languages [ 19 ].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Using interactive Jupyter Notebooks and BioConda for FAIR and reproducible biomolecular simulation workflows

Bayarri,

Andrio,

Gelpí

et al. 2024

PLoS Comput Biol

Self Cite

View full text Add to dashboard Cite

Interactive Jupyter Notebooks in combination with Conda environments can be used to generate FAIR (Findable, Accessible, Interoperable and Reusable/Reproducible) biomolecular simulation workflows. The interactive programming code accompanied by documentation and the possibility to inspect intermediate results with versatile graphical charts and data visualization is very helpful, especially in iterative processes, where parameters might be adjusted to a particular system of interest. This work presents a collection of FAIR notebooks covering various areas of the biomolecular simulation field, such as molecular dynamics (MD), protein–ligand docking, molecular checking/modeling, molecular interactions, and free energy perturbations. Workflows can be launched with myBinder or easily installed in a local system. The collection of notebooks aims to provide a compilation of demonstration workflows, and it is continuously updated and expanded with examples using new methodologies and tools.

show abstract

Section: Discussion/conclusionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using interactive Jupyter Notebooks and BioConda for FAIR and reproducible biomolecular simulation workflows

Bayarri,

Andrio,

Gelpí

et al. 2024

PLoS Comput Biol

Self Cite

View full text Add to dashboard Cite

show abstract

“…Galaxy is developing support for importing and exporting Workflow Run Crates, a profile of RO-Crate (Soiland-Reyes et al 2022b) to captures execution history of a workflow, including its definition and intermediate data (De Geest et al 2022). SDR is adopting this support to combine openDS FDOs with workflow provenance, as envisioned by Walton et al (2020).…”

Section: Ro-crate Lessonsmentioning

confidence: 99%

Incrementally building FAIR Digital Objects with Specimen Data Refinery workflows

Woolland¹,

Brack²,

Soiland‐Reyes³

et al. 2022

RIO

Self Cite

View full text Add to dashboard Cite

Specimen Data Refinery (SDR) is a developing platform for automating transcription of specimens from natural history collections (Hardisty et al. 2022). SDR is based on computational workflows and digital twins using FAIR Digital Objects. We show our recent experiences with building SDR using the Galaxy workflow system and combining two FDO methodologies with open digital specimens (openDS) and RO-Crate data packaging. We suggest FDO improvements for incremental building of digital objects in computational workflows. SDR workflows SDR is realised as the workflow system Galaxy (Afgan et al. 2018) with SDR tools installed. An Open Research challenge is that some tools have machine learning models with a commercial licence. This complicates publishing to Galaxy toolshed, however we created Ansible scripts to install equivalent Galaxy servers, including tools and dependencies, accounts and workflows. SDR workflows are published in WorkflowHub as FDOs. We implemented the use case De novo digitization in Galaxy (Brack et al. 2022). Shown in Fig. 1 the workflow steps exchange openDS JSON (Hardisty et al. 2019), for incremental completion of a digital specimen. Initial stages build a template openDS from a CSV with metadata and image references – subsequent analysis completes the rest of the JSON with regions of interest, text digitised from handwriting, and recognized named entities. Galaxy can visualise outputs of each step (Fig. 2), important to make the FDOs understandable by domain experts and to verify accuracy of SDR. We are adding workflows for partial stages, e.g. detection of regions (Livermore and Woolland 2022a) and hand-written text recognition (Livermore and Woolland 2022b), which we'll combine with scalability testing and wider testing by project users. Additional workflows will enhance existing FDOs and use new tools such as barcode detection of museums’ internal identifiers. We are now ready to publish digital specimens as FAIR Digital Objects, with registration into DiSSCO repositories, PID assignment and workflow provenance. However, even at this early stage we have identified several challenges that need to be addressed. FDO lessons We highlight the De novo use case because this workflow is exchanging partial FDOs – openDS objects which are not fully completed and not yet assigned persistent identifiers. openDS schemas are still in development, therefore SDR uses a more flexible JSON schema where only the initial metadata (populated from CSV) are required. Each step validates the partial FDO before passing it to the underlying command line tool. Although workflow steps exchange openDS objects, they cannot be combined in any order. For instance, named entity recognition requires digitised text in the FDO. We can consider these intermediate steps as sub-profiles of an FDO Type. Unlike hierarchical subclasses, these FDO profiles are more like ducktyping. For instance a text detection step may only require the regions key, but semantically there is no requirement for an OpenDSWithText to be a subclass of OpenDSWithRegion, as text also can be transcribed manually without regions. Similarly, we found that some steps can be executed in parallel, but this requires merging of partial FDOs. This can be achieved by combining JSON queries and JSON Schemas, but indicates that it may be more beneficial to have FDO fragments as separate objects. Adding openDS fragment steps would however complicate workflows. Several of our tools process the referenced images, currently https URLs in openDS. We added a caching layer to avoid repeated image downloading, coupled with local file-paths wiring in the workflow. A similar challenge occurs if accessing image data using DOIP, which unlike HTTP, has no caching mechanisms. RO-Crate lessons Galaxy is developing support for importing and exporting Workflow Run Crates, a profile of RO-Crate (Soiland-Reyes et al. 2022b) to captures execution history of a workflow, including its definition and intermediate data (De Geest et al. 2022). SDR is adopting this support to combine openDS FDOs with workflow provenance, as envisioned by Walton et al. (2020). Our prototype de novo workflow returns results as a ZIP file of openDS objects. End-users should also get copies of the referenced images and generated visualisations, along with workflow execution metadata. We are investigating ways to embed the preliminary Galaxy workflow history before the final step, so that this result can be an enriched RO-Crate. Conclusions SDR is an example of machine-assisted construction of FDOs, which highlight the needs for intermediate digital objects that are not yet FDO compliant. The passing of such “local FDOs” is beneficial not just for efficiency and visual inspection, but also to simplify workflow composition of canonical workflow building blocks. At the same time we see that it is insufficient to only pass FDOs as JSON objects, as they also have references to other data such as images, which should not need to be re-downloaded. Further work will investigate the use of RO-Crate as a wrapper of partial FDOs, but this needs to be coupled with more flexible FDO types as profiles, in order to restrict “impossible” ordering of steps depending on particular inner FDO fragments. A distinction needs to be made between open digital specimens that are in “draft” state and those that can be pushed to DiSSCo registries. We are experimenting with changing the SDR components into Canonical Workflow Building Blocks (Soiland-Reyes et al. 2022a) using the Common Workflow Language (Crusoe et al. 2022). This gives flexibility to scalably execute SDR workflows on different compute backends such as HPC or local cluster, without the additional setup of Galaxy servers.

show abstract

Research and Application of Optimal Value Calculation Method for Mixed Data Blocks based on Dynamic Programming Algorithm

Chen,

Bao

et al. 2024

2024 International Conference on Integrated Circuits and Communication Systems (ICICACS)

View full text Add to dashboard Cite

Making Canonical Workflow Building Blocks Interoperable across Workflow Languages

Cited by 4 publications

References 31 publications

Using interactive Jupyter Notebooks and BioConda for FAIR and reproducible biomolecular simulation workflows

Using interactive Jupyter Notebooks and BioConda for FAIR and reproducible biomolecular simulation workflows

Incrementally building FAIR Digital Objects with Specimen Data Refinery workflows

Research and Application of Optimal Value Calculation Method for Mixed Data Blocks based on Dynamic Programming Algorithm

Contact Info

Product

Resources

About