Why-Diff: Exploiting Provenance to Understand Outcome Differences From Non-Identical Reproduced Workflows

Abstract

Data analytics processes such as scientific workflows tend to be executed repeatedly, with varying dependencies and input datasets. The case has been made in the past for tracking the provenance of the final information products through the workflow steps, to enable their reproducibility. In this paper, we explore the hypothesis that provenance traces recorded during execution are also instrumental to answering questions about the observed differences between sets of results obtained from similar but not identical workflow configurations. Such differences in configurations may be introduced deliberately, i.e., to explore process variations or accidentally, typically as the result of porting efforts or of changes in the computing environment. Using a commonly used workflow programming model as a reference, we consider both structural variations in the workflows as well as variations within their individual components. Our why-diff algorithm compares the graph representations of two provenance traces derived from two workflow variations. It produces a delta graph that can be used to produce human-readable explanations of the impact of workflow differences on observed output differences. We report with Neo4j graph database. Further, we report explanations of the difference between workflow results using a suite of synthetic workflows as well as real-world workflows. INDEX TERMS Why-diff, provenance, reproducibility, big data, eScience Central, workflow.

Capturing provenance information for biomedical data and workflows: A scoping review

“…They also provide detailed characteristics about the respective provenance approaches. 2009-2011 6 [10], [29], [30], [31], [32], [33] 2012-2014 13 [1], [2], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44] 2015-2017 9 [4], [45], [46], [47], [48], [49], [50], [51], [52] 2018-2020 21 [9], [11], [53], [54], [55] [70], [71] Target domain (Bio-) medical or healthcare domain 36 [1], [2], [4], [9], [10], [11], [28], [29], [31], [34], [36], [37], [38], [39], [40], [42], [43], [44], [46], [47], [48], [49], [51]…”

Making Provenance Work for You

“…Provenance collection is common in workflow systems where it is built directly into the execution environment, such as in Kepler (Altintas et al, 2006), VisTrails (Koop et al, 2013), and Taverna (Missier et al, 2008). Of particular interest is the work of de Oliveira et al (2014) who use provenance to debug long-running workflows, and Why-Diff (Thavasimani et al, 2019) which compares provenance of multiple workflow executions to find differences. Provenance collection in programming languages is much less common, with the exception of the noWorkflow (Murta et al, 2014) implementation for Python.…”

Square Kilometre Array : Processing Voluminous MeerKAT Data on IRIS

“…Execution provenance helps to understand the results of reproduced workflows. Extracting and understanding the provenance for tracking the changes in the reproduced workflows is widely discussed and demonstrated in the papers [23] and [24].…”