Software architectures to integrate workflow engines in science gateways

Abstract

International audienceScience gateways often rely on workflow engines to execute applications on distributed infrastructures. We investigate six software architectures commonly used to integrate workflow engines into science gateways. In tight integration, the workflow engine shares software components with the science gateway. In service invocation, the engine is isolated and invoked through a specific software interface. In task encapsulation, the engine is wrapped as a computing task executed on the infrastructure. In the pool model, the engine is bundled in an agent that connects to a central pool to fetch and execute workflows. In nested workflows, the engine is integrated as a child process of another engine. In workflow conversion, the engine is integrated through workflow language conversion. We describe and evaluate these architectures with metrics for assessment of integration complexity, robustness, extensibility, scalability and functionality. Tight integration and task encapsulation are the easiest to integrate and the most robust. Extensibility is equivalent in most architectures. The pool model is the most scalable one and meta-workflows are only available in nested workflows and workflow conversion. These results provide insights for science gateway architects and developers

Exploiting GPUs on distributed infrastructures for medical imaging applications with VIP and DIRAC

“…The Virtual Imaging Platform [1], [8] (VIP 1 ) is a web portal for medical simulation and image data analysis. It leverages resources available in the Biomed VO of the EGI e-infrastructure to offer an open service to academic researchers worldwide.…”

Scientific workflows: Past, present and future

“…In the paper "Software architectures to integrate workflow engines in science" [18], Glatard provides a comparison of six different software architectures commonly used to integrate workflow engines into science gateways. By analysing these architectures, the authors extract several different types of components and interactions and provide a set of metrics that can be used to rate the architectures, including complexity, robustness, extensibility, scalability, and the support for meta-workflows as well as fine-grained debugging.…”

“…For example, improvements in engineering will yield responsiveness that helps usability, whereas, the verifiable rule enforcement needed to achieve sustained collaboration across autonomous organisations will incur costs. Improved architectures will mitigate these effects, as described in one paper in this issue [18]. Improvements in theory underpinning workflow languages and systems, such as the work of Wadler 12 and Cheney [31], will become more essential as complexity and scale grow.…”

“…The data collected across the IoT landscape from devices is vast. According to Cisco, all of the people and things connected to the Internet will generate 507.5 zettabytes of data by 2019 18 . Sending all this information to centralised servers will lead to bottlenecks, increase latency and be inefficient, especially when some or all of processing can be computed near the source(s).…”

MULTI-X, a State-of-the-Art Cloud-Based Ecosystem for Biomedical Research

“…They are massively used in their specific domains, although they often lack flexible models to integrate new components, or solutions for scalable computational resources. With a similar approach, Science Gateways [14] such as CBRAIN, NSG, CSGF, WS-PGRADE/gUSE, and Globus Galaxies, have been deployed to integrate efforts and share existing resources. These systems allow programmatic access to external data sources and include one or several workflow engines.…”