Crossing analytics systems: A case for integrated provenance in data lakes

Suriarachchi, Isuru; Plale, Beth

doi:10.1109/escience.2016.7870919

Cited by 33 publications

(60 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The integration of this information into the metadata system makes it possible to understand and explain possible inconsistencies in the data [3]. It can also be used to manage sensitive data, by detecting intrusions [26].…”

Section: Expected Featuresmentioning

confidence: 99%

“…Thus, such features are often integrated together in a provenance tracking module [12,13,27]. Yet, we still consider that they remain different features since they are not systematically proposed together [3,5,26].…”

Section: Expected Featuresmentioning

confidence: 99%

“…A data lake provides integrated data storage without predefined schema [11]. In the absence of a data schema, an effective metadata system becomes essential to make data queryable and thus prevent the lake from turning into a data swamp, i.e., an inexploitable data lake [1,11,26].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Metadata Systems for Data Lakes: Models and Features

Sawadogo

Scholly

Favre

et al. 2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

2 BIAL-X https://www.bial-x.com/ Abstract. Over the past decade, the data lake concept has emerged as an alternative to data warehouses for storing and analyzing big data. A data lake allows storing data without any predefined schema. Therefore, data querying and analysis depend on a metadata system that must be efficient and comprehensive. However, metadata management in data lakes remains a current issue and the criteria for evaluating its effectiveness are more or less nonexistent. In this paper, we introduce MEDAL, a generic, graph-based model for metadata management in data lakes. We also propose evaluation criteria for data lake metadata systems through a list of expected features. Eventually, we show that our approach is more comprehensive than existing metadata systems.

show abstract

Section: Expected Featuresmentioning

confidence: 99%

Section: Expected Featuresmentioning

confidence: 99%

See 1 more Smart Citation

Metadata Systems for Data Lakes: Models and Features

Sawadogo

Scholly

Favre

et al. 2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…This is known as schema-on-read or late binding (Fang, 2015;Miloslavskaya and Tolstoy, 2016). However, with big data volume and velocity coming into play, the absence of an explicit schema can quickly turn a data lake into an inoperable data swamp (Suriarachchi and Plale, 2016). Therefore, metadata management is a crucial component in data lakes (Quix et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

Metadata Management for Textual Documents in Data Lakes

Sawadogo

Kibata

Darmont

2019

Proceedings of the 21st International Conference on Enterprise Information Systems

View full text Add to dashboard Cite

Data lakes have emerged as an alternative to data warehouses for the storage, exploration and analysis of big data. In a data lake, data are stored in a raw state and bear no explicit schema. Thence, an efficient metadata system is essential to avoid the data lake turning to a so-called data swamp. Existing works about managing data lake metadata mostly focus on structured and semi-structured data, with little research on unstructured data. Thus, we propose in this paper a methodological approach to build and manage a metadata system that is specific to textual documents in data lakes. First, we make an inventory of usual and meaningful metadata to extract. Then, we apply some specific techniques from the text mining and information retrieval domains to extract, store and reuse these metadata within the COREL research project, in order to validate our proposals.

show abstract

“…For systems that generate heterogeneous provenance data, such as in data lakes [44,45], standard representations can help reduce the complexity of integrating such provenance into a common model [46]. ABM relationships captured by ProvMASS are specified with concepts from the W3C PROV [47].…”

Section: Provenance-related Propertiesmentioning

confidence: 99%

Data Provenance for Agent-Based Models in a Distributed Memory

Davis

Featherston

et al. 2018

Informatics

View full text Add to dashboard Cite

Agent-Based Models (ABMs) assist with studying emergent collective behavior of individual entities in social, biological, economic, network, and physical systems. Data provenance can support ABM by explaining individual agent behavior. However, there is no provenance support for ABMs in a distributed setting. The Multi-Agent Spatial Simulation (MASS) library provides a framework for simulating ABMs at fine granularity, where agents and spatial data are shared application resources in a distributed memory. We introduce a novel approach to capture ABM provenance in a distributed memory, called ProvMASS. We evaluate our technique with traditional data provenance queries and performance measures. Our results indicate that a configurable approach can capture provenance that explains coordination of distributed shared resources, simulation logic, and agent behavior while limiting performance overhead. We also show the ability to support practical analyses (e.g., agent tracking) and storage requirements for different capture configurations.

show abstract

Crossing analytics systems: A case for integrated provenance in data lakes

Cited by 33 publications

References 13 publications

Metadata Systems for Data Lakes: Models and Features

Metadata Systems for Data Lakes: Models and Features

Metadata Management for Textual Documents in Data Lakes

Data Provenance for Agent-Based Models in a Distributed Memory

Contact Info

Product

Resources

About