When History Matters - Assessing Reliability for the Reuse of Scientific Workflows

Abstract

Abstract. Scientific workflows play an important role in computational research as essential artifacts for communicating the methods used to produce research findings. We are witnessing a growing number of efforts that treat workflows as first-class artifacts for sharing and exchanging scientific knowledge, either as part of scholarly articles or as stand-alone objects. However, workflows are not born to be reliable, which can seriously damage their reusability and trustworthiness as knowledge exchange instruments. Scientific workflows are commonly subject to decay, which consequently undermines their reliability over their lifetime. The reliability of workflows can be notably improved by advocating scientists to preserve a minimal set of information that is essential to assist the interpretations of these workflows and hence improve their potential for reproducibility and reusability. In this paper we show how, by measuring and monitoring the completeness and stability of scientific workflows over time we are able to provide scientists with a measure of their reliability, supporting the reuse of trustworthy scientific knowledge.

A Quantitative Study on the Re-executability of Publicly Shared Scientific Workflows

“…Challenges in workflow reproducibility stemming from the dependency to third-party services are discussed in [9], [10]. In [11], the authors investigate the workflow decay not in a specific moment in time, but over a period of time, including the impact of workflow evolution.…”

A Domain-Adaptable Heterogeneous Information Integration Platform: Tourism and Biomedicine Domains

“…Keeping data up to date and the maintenance of consistency and traceability are ever-harder challenges. In this context, the quality of an information system can be measured in terms of several dimensions, including originality, accuracy [58], completeness, and reliability [59,60]. Measurements can be made using formal or informal methods.…”

Enabling FAIR research in Earth Science through research objects

“…Research objects with high quality metadata are more likely to be reused than low quality ones, and in the long term such quality could experience changes, for example when some input file (e.g, an annotation file) becomes unavailable, degrading the overall quality of the research object and introducing decay. Inspired in wet lab practices checklists [27] were proposed as the main tool to assess the quality of research objects through their lifecycle [28]. These checklists are made up of statements that specify the required metadata a research object must contain.…”

“…Checklists collect the necessary information to calculate quality metrics about the completeness, stability and reliability of research objects [28]. Completeness measures the extent to which a research object satisfies a number of requirements specified in a checklist, stability measures the degree to which the research object completeness remains unchanged, and reliability combines both previous metrics to provide a unique value indicating to what extent the research object is complete and how stable it has been historically.…”