Komadu: A Capture and Visualization System for Scientific Data Provenance

Abstract

IntroductionData provenance is information about the entities, activities and people who have effected some type of transformation on a data product through the product's lifecycle. Data provenance captured from scientific applications is a critical precursor to data sharing and reuse. For researchers wanting to repurpose and reuse data, it is a source of information about the lineage and attribution of the data and this is needed in order to establish trust in a data set. Data provenance has been shown useful in results validation, failure tracing, and reproducibility. The Komadu provenance capture system is standalone, meaning it is not coupled to or dependent upon any database management system, repository, or scientific workflow system. It provides an ingest API through which provenance notifications are fed into the system at high speeds, and a query API through which provenance information can be queried. The data model is both event oriented and graph oriented, in that graphs are pieced together in Komadu based on the events received from the environment.Komadu has its roots in the Karma [2] provenance capture system, an earlier version that complied with the OPM community standard [3] both for defining the type of provenance notifications that the system accepted, and for defining the format of the results. Komadu, on the other hand, supports the W3C PROV specification [1] which provides far richer types of relationships and has a more formal model for handling time than does OPM. Karma was additionally limited by assuming that every notification belonging to the same external activity shared a common global identifier that is shared across all components (services, methods etc.) of the external environment. This limitation was found to be severe in applications where provenance is not only captured at the application level, but also at in the larger environment where the application runs. Take for instance a distributed application running in PlanetLab [7] and running under Twister [8]; it is highly limiting to expect provenance events generated from the application, from PlanetLab, and from Twister to all have shared knowledge about any single global identifier. This limitation derives from Karma's early days where it tracked provenance for applications running within a single workflow system. Additionally, a researcher may be interested in tracking lineage starting from some data product or agent. Such scenarios are not supported by Karma.In this paper, we introduce Komadu [9] provenance capture and visualization system. Komadu is a complete redesign and reimplementation of Karma that supports new features while addressing the above mentioned limitations of Karma. The main contributions of Komadu are as follows. . Even though Komadu has been used most extensively in relation to scientific research, its interfaces are designed to collect and visualize provenance of any kind of application needing provenance.

Detecting misinformation in social networks using provenance data

“…DAX files represent the abstract description of a single workflow in the XML format. The provenance recorded from the logs of the simulation were generated using ProvToolBox and input into KOMADU, a standalone provenance capture and visualization system for capturing, representing, and manipulating provenance. KOMADU uses the W3C PROV standard, considered a successor to the Karma provenance capture system.…”

Application of provenance in social computing: A case study

“…In addition to the difference that Komadu is based on PROVO, while Karma on OPM, Komadu has addressed several limitations of Karma, such as upscaling provenance relationship types as well as enabling provenance data capturing from different sources and applications as provenance data in Komadu does not need to be connected to a single identifier like in Karma. Events in the form of notifications can be captured through Komadu's inject API, and data can be queried through its query API …”

“…Events in the form of notifications can be captured through Komadu's inject API, and data can be queried through its query API. 26 We deployed Komadu, with Tomcat 7 and MySQL 5.5 on a server of the t2.micro model † from Amazon. The experiment we conducted consists in evaluating the type of data that can be modeled in Komadu and the processing time for provenance data-injection into it.…”

Provenance threat modeling

“…In application to digital scientific data, provenance is an important component in broadening, sharing, and reusing scientific data [4]. Provenance is useful in validating results, failure tracing, and reproducibility [5].…”