A survey on provenance: What for? What form? What from?

Herschel, Melanie; Diestelkämper, Ralf; Lahmar, Houssem Ben

doi:10.1007/s00778-017-0486-1

Cited by 222 publications

(182 citation statements)

References 147 publications

Supporting

Mentioning

178

Contrasting

Unclassified

Order By: Relevance

“…Provenance [27,53,64,142] is likely to be a key element in assisting the user in choosing a dataset of interest. Until now, provenance has been used to facilitate trust in an artifact [37,38] or automatically estimate quality [66].…”

Section: Database Building Blocksmentioning

confidence: 99%

“…New methods must be developed to facilitate translation of this large graph into a format that a user who is evaluating whether or not to use a dataset can interpret and utilize [29]. The logic and possible new operators behind dataset search will open up new areas for determining why and why not to consider provenance of the dataset query results themselves [30,64,87].…”

Section: Database Building Blocksmentioning

confidence: 99%

“…These limitations impact the use of the retrieved data -machine learning can be unduly affected by the processing that was performed over a dataset prior to its release [125], while knowing the original purpose for collecting the data aids interpretation and analysis [140]. In other words, in a dataset search context, approaches need to consider additional aspects such as data provenance [27,53,64,87,101,142], annotations [67,93,144], quality [116,131,148], granularity of content [81], and schema [9,20] to effectively evaluate a dataset's fitness for a particular use. The user does not have the ability to introspect over large amounts of data, and their attention must be prioritized [13].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dataset search: a survey

et al. 2019

View full text Add to dashboard Cite

Generating value from data requires the ability to find, access and make sense of datasets. There are many efforts underway to encourage data sharing and reuse, from scientific publishers asking authors to submit data alongside manuscripts to data marketplaces, open data portals and data communities. Google recently beta released a search service for datasets, which allows users to discover data stored in various online repositories via keyword queries. These developments foreshadow an emerging research field around dataset search or retrieval that broadly encompasses frameworks, methods and tools that help match a user data need against a collection of datasets. Here, we survey the state of the art of research and commercial systems in dataset retrieval. We identify what makes dataset search a research field in its own right, with unique challenges and methods and highlight open problems. We look at approaches and implementations from related areas dataset search is drawing upon, including information retrieval, databases, entity-centric and tabular search in order to identify possible paths to resolve these open problems as well as immediate next steps that will take the field forward.

show abstract

Section: Database Building Blocksmentioning

confidence: 99%

Section: Database Building Blocksmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dataset search: a survey

et al. 2019

View full text Add to dashboard Cite

show abstract

“…The workflows are instrumented with PLLib, imported as a library in the scripts, which is responsible for the provenance data capture. In an offline manner and following the methodology described in a previous work [17] to specify the workflows using prospective provenance data standards [8], users add provenance data capture calls using the PLLib. A provenance capture task happens when a data transformation executes, which typically occurs in a function call, in a…”

Section: Provlake In the ML Lifecycle In Csementioning

confidence: 99%

Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering

Souza¹,

Valduriez²,

Mattoso³

et al. 2019

2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)

View full text Add to dashboard Cite

Machine Learning (ML) has become essential in several industries. In Computational Science and Engineering (CSE), the complexity of the ML lifecycle comes from the large variety of data, scientists' expertise, tools, and workflows. If data are not tracked properly during the lifecycle, it becomes unfeasible to recreate a ML model from scratch or to explain to stakeholders how it was created. The main limitation of provenance tracking solutions is that they cannot cope with provenance capture and integration of domain and ML data processed in the multiple workflows in the lifecycle, while keeping the provenance capture overhead low. To handle this problem, in this paper we contribute with a detailed characterization of provenance data in the ML lifecycle in CSE; a new provenance data representation, called PROV-ML, built on top of W3C PROV and ML Schema; and extensions to a system that tracks provenance from multiple workflows to address the characteristics of ML and CSE, and to allow for provenance queries with a standard vocabulary. We show a practical use in a real case in the O&G industry, along with its evaluation using 48 GPUs in parallel.Index Terms-Machine Learning Lifecycle, Workflow Provenance, Computational Science and Engineering (ii) PROV-ML: a new data representation, which combines W3C PROV [18] with W3C ML Schema [19], for prove-R. Souza et al. Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering.

show abstract

“…In prior work, we have shown that why and why-not provenance can be treated uniformly as provenance for firstorder (FO) queries (= non-recursive Datalog with negation) [17] and have implemented this idea in a system called PUG (Provenance Unification through Graphs) [20,21]. Instancebased [12] why-not provenance techniques determine which existing and missing data from a query's input is responsible for the failure to derive a missing answer of interest. These techniques either (i) enumerate all potential ways of deriving a result (all-derivations approach) or (ii) return only one possible, but failed, derivation or parts thereof (singlederivation approach).…”

Section: Introductionmentioning

confidence: 99%

Approximate summaries for why and why-not provenance

2020

View full text Add to dashboard Cite

Why and why-not provenance have been studied extensively in recent years. However, why-not provenance and-to a lesser degree-why provenance, can be very large resulting in severe scalability and usability challenges. In this paper, we introduce a novel approximate summarization technique for provenance which overcomes these challenges. Our approach uses patterns to encode (why-not) provenance concisely. We develop techniques for efficiently computing provenance summaries balancing informativeness, conciseness, and completeness. To achieve scalability, we integrate sampling techniques into provenance capture and summarization. Our approach is the first to scale to large datasets and to generate comprehensive and meaningful summaries. PVLDB Reference Format:Seokki Lee, Bertram Ludäscher, Boris Glavic. Approximate Summaries for Why and Why-not Provenance. PVLDB, 13(6): xxxxyyyy, 2020.

show abstract

A survey on provenance: What for? What form? What from?

Cited by 222 publications

References 147 publications

Dataset search: a survey

Dataset search: a survey

Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering

Approximate summaries for why and why-not provenance

Contact Info

Product

Resources

About