Multi-level meta-workflows: new concept for regularly occurring tasks in quantum chemistry

Abstract

Background In Quantum Chemistry, many tasks are reoccurring frequently, e.g. geometry optimizations, benchmarking series etc. Here, workflows can help to reduce the time of manual job definition and output extraction. These workflows are executed on computing infrastructures and may require large computing and data resources. Scientific workflows hide these infrastructures and the resources needed to run them. It requires significant efforts and specific expertise to design, implement and test these workflows.SignificanceMany of these workflows are complex and monolithic entities that can be used for particular scientific experiments. Hence, their modification is not straightforward and it makes almost impossible to share them. To address these issues we propose developing atomic workflows and embedding them in meta-workflows. Atomic workflows deliver a well-defined research domain specific function. Publishing workflows in repositories enables workflow sharing inside and/or among scientific communities. We formally specify atomic and meta-workflows in order to define data structures to be used in repositories for uploading and sharing them. Additionally, we present a formal description focused at orchestration of atomic workflows into meta-workflows.ConclusionsWe investigated the operations that represent basic functionalities in Quantum Chemistry, developed the relevant atomic workflows and combined them into meta-workflows. Having these workflows we defined the structure of the Quantum Chemistry workflow library and uploaded these workflows in the SHIWA Workflow Repository.Graphical AbstractMeta-workflows and embedded workflows in the template representation Electronic supplementary materialThe online version of this article (doi:10.1186/s13321-016-0169-8) contains supplementary material, which is available to authorized users.

Scientific workflows: Past, present and future

“…A coarse-grained workflow interoperability approach that enables embedding non-native workflows into a hosting meta-workflow has been described by Terstysanszky et al [47]. The importance of such metaworkflows has been demonstrated in Herres-Pawlis et al [48] by combining heterogeneous workflow systems to build meta-workflows for computational chemistry. WS-PGRADE/gUSE is widely used via domain-specific science gateways -over 30 have been built in the SCI-BUS project [49].…”