Scientific workflows: Past, present and future

Abstract

International audienceThis special issue and our editorial celebrate 10 years of progress with data-intensive or scientific workflows. There have been very substantial advances in the representation of workflows and in the engineering of workflow management systems (WMS). The creation and refinement stages are now well supported, with a significant improvement in usability. Improved abstraction supports cross-fertilisation between different workflow communities and consistent interpretation as WMS evolve. Through such re-engineering the WMS deliver much improved performance, significantly increased scale and sophisticated reliability mechanisms. Further improvement is anticipated from substantial advances in optimisation. We invited papers from those who have delivered these advances and selected 14 to represent today's achievements and representative plans for future progress. This editorial introduces those contributions with an overview and categorisation of the papers. Furthermore, it elucidates responses from a survey of major workflow systems, which provides evidence of substantial progress and a structured index of related papers. We conclude with suggestions on areas where further research and development is needed and offer a vision of future research directions

Establishing Core Concepts for Information-Powered Collaborations

“…Scientific methods support -helps collaborating teams of experts create and refine methods that draw on the diverse resources and data collections. It promotes the formalisation and automation of these methods, typically as scientific workflows [10], while supporting critical procedures to deliver good quality evidence contributing to the shared knowledge.…”

The Fourth Paradigm 10 Years On

“…Computer Systems [17] follows developments of workflow systems over the last decade. Workflow systems have clearly advanced substantially in ease of use, support for improved abstractions, automation of data identification, and inbuilt tools for provenance.…”

MULTI-X, a State-of-the-Art Cloud-Based Ecosystem for Biomedical Research

“…Workflow Management. The automation of the execution of computational tasks is a key factor to improve scientific productivity [8]. Workflows provide the interface between scientist and complex computing infrastructures.…”