Executing cyclic scientific workflows in the cloud

Abstract

We present an algorithm and a software architecture for a cloud-based system that executes cyclic scientific workflows whose structure may change during run time. Existing approaches either rely on workflow definitions based on directed acyclic graphs (DAGs) or require workarounds to implement cyclic structures. In contrast, our system supports cycles natively, avoids workarounds, and as such reduces the complexity of workflow modelling and maintenance. Our algorithm traverses workflow graphs and transforms them iteratively into linear sequences of executable actions. We call these sequences process chains. Our software architecture distributes the process chains to multiple compute nodes in the cloud and oversees their execution. We evaluate our approach by applying it to two practical use cases from the domains of astronomy and engineering. We also compare it with two existing workflow management systems. The evaluation demonstrates that our algorithm is able to execute dynamically changing workflows with cycles and that design and maintenance of complex workflows is easier than with existing solutions. It also shows that our software architecture can run process chains on multiple compute nodes in parallel to significantly speed up the workflow execution. An implementation of our algorithm and the software architecture is available with the Steep Workflow Management System that we released under an open-source license. The resources for the first practical use case are also available as open source for reproduction.

A dynamic knowledge graph approach to distributed self-driving laboratories

“…Each goal is associated with specified dimensional quantities that can be achieved by a Plan, which consists of multiple Steps to be carried out by corresponding agents. From the implementation perspective, this is akin to a specialised research sub-domain within the scientific workflow community that focuses on the management of iterative workflows abstracted as directed cyclic graphs 46 . In this regard, we adopt the derived information framework 42 , a knowledge-graph-native approach, to manage the iterative workflow.…”

From Platform to Knowledge Graph: Distributed Self-Driving Laboratories

“…Each goal is associated with specific dimensional quantities that can be achieved by a Plan, which consists of multiple Steps to be carried out by corresponding agents. From the implementation perspective, this is akin to research in the scientific workflow community which seeks to manage workflow expressed as a directed cyclic graph (DCG) [44]. In this regard, we adopt the derived information framework [40], a knowledge-graph-native approach, to manage the iterative workflow.…”

A Derived Information Framework for a Dynamic Knowledge Graph and its Application to Smart Cities

“…We emphasise that the primary intent of the derivation framework is to represent logical dependencies as such, as a historical record, rather than consider them as 'steps' in a workflow or algorithm. This implies that, in contrast to workflows [55], cyclic dependencies are not permitted in the derivation framework, as they would amount to logical contradictions. Nonetheless, the framework can of course be used to record the dependencies of pieces of information that were obtained from algorithms containing loops and other circular constructs.…”