Uncovering the evolution history of data lakes

Klettke, Meike; Awolin, Hannes; Störl, Uta; Müller, D.; Scherzinger, Stefanie

doi:10.1109/bigdata.2017.8258204

Cited by 24 publications

(12 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Singh et al show that Bayesian models allow detecting links between data attributes [50]. Similarly, several authors propose algorithms to discover schemas or constraints in semistructured data [3,27,44].…”

Section: Link_documentmentioning

confidence: 99%

On data lake architectures and metadata management

2020

View full text Add to dashboard Cite

Over the past two decades, we have witnessed an exponential increase of data production in the world. So-called big data generally come from transactional systems, and even more so from the Internet of Things and social media. They are mainly characterized by volume, velocity, variety and veracity issues. Big data-related issues strongly challenge traditional data management and analysis systems. The concept of data lake was introduced to address them. A data lake is a large, raw data repository that stores and manages all company data bearing any format. However, the data lake concept remains ambiguous or fuzzy for many researchers and practitioners, who often confuse it with the Hadoop technology. Thus, we provide in this paper a comprehensive state of the art of the different approaches to data lake design. We particularly focus on data lake architectures and metadata management, which are key issues in successful data lakes. We also discuss the pros and cons of data lakes and their design alternatives.

show abstract

Section: Link_documentmentioning

confidence: 99%

On data lake architectures and metadata management

2020

View full text Add to dashboard Cite

show abstract

“…For the task of schema evolution management, we use Darwin to initially declare or extract schemas, define schema evolution operations, or extract schema versions and schema evolution operations from legacy data. We have presented these functionalities in Klettke et al [7] and Störl et al [14]. In this present paper, we present a methodology of self-adapting data migration which builds on our demo paper about Mig-Cast [4], which focuses on data migration itself.…”

Section: Architecturementioning

confidence: 99%

Self-adapting data migration in the context of schema evolution in NoSQL databases

Hillenbrand

Störl

Nabiyev

et al. 2021

Distrib Parallel Databases

Self Cite

View full text Add to dashboard Cite

When NoSQL database systems are used in an agile software development setting, data model changes occur frequently and thus, data is routinely stored in different versions. The management of versioned data leads to an overhead potentially impeding the software development. Several data migration strategies exist that handle legacy data differently during data accesses, each of which can be characterized by certain advantages and disadvantages. Depending on the requirements for the software application, we evaluate and compare different migration strategies through metrics like migration costs and latency as well as precision and recall. Ideally, exactly that strategy should be selected whose characteristics fulfill service-level agreements and match the migration scenario, which depends on the query workload and the changes in the data model which imply an evolution of the database schema. In this paper, we present a methodology of self-adapting data migration, which automatically adjusts migration strategies and their parameters with respect to the migration scenario and service-level agreements, thereby contributing to the self-management of database systems and supporting agile development.

show abstract

“…Thus, to ensure data accessibility, exploration, and exploitation, an efficient and effective metadata system becomes an indispensible component in data lakes (Quix et al, 2016). Yet, most of the research work on data lakes still concentrate on structured data, or semi-structured data only (Farid et al, 2016;Farrugia et al, 2016;Madera and Laurent, 2016;Quix et al, 2016;Klettke et al, 2017). So far, unstructured data have not received enough consideration in the relevant research literature, while more often than not unstructured heterogeneous data occur frequently (Miloslavskaya and Tolstoy, 2016).…”

Section: Related Workmentioning

confidence: 99%

On the Logical Design of a Prototypical Data Lake System for Biological Resources

Che

Duan

2020

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

Biological resources are multifarious encompassing organisms, genetic materials, populations, or any other biotic components of ecosystems, and fine-grained data management and processing of these diverse types of resources proposes a tremendous challenge for both researchers and practitioners. Before the conceptualization of data lakes, former big data management platforms in the research fields of computational biology and biomedicine could not deal with many practical data management tasks very well. As an effective complement to those previous systems, data lakes were devised to store voluminous, varied, and diversely structured or unstructured data in their native formats, for the sake of various analyses like reporting, modeling, data exploration, knowledge discovery, data visualization, advanced analysis, and machine learning. Due to their intrinsic traits, data lakes are thought to be ideal technologies for processing of hybrid biological resources in the format of text, image, audio, video, and structured tabular data. This paper proposes a method for constructing a practical data lake system for processing multimodal biological data using a prototype system named ProtoDLS, especially from the explainability point of view, which is indispensable to the rigor, transparency, persuasiveness, and trustworthiness of the applications in the field. ProtoDLS adopts a horizontal pipeline to ensure the intra-component explainability factors from data acquisition to data presentation, and a vertical pipeline to ensure the inner-component explainability factors including mathematics, algorithm, execution time, memory consumption, network latency, security, and sampling size. The dual mechanism can ensure the explainability guarantees on the entirety of the data lake system. ProtoDLS proves that a single point of explainability cannot thoroughly expound the cause and effect of the matter from an overall perspective, and adopting a systematic, dynamic, and multisided way of thinking and a system-oriented analysis method is critical when designing a data processing system for biological resources.

show abstract

Uncovering the evolution history of data lakes

Cited by 24 publications

References 11 publications

On data lake architectures and metadata management

On data lake architectures and metadata management

Self-adapting data migration in the context of schema evolution in NoSQL databases

On the Logical Design of a Prototypical Data Lake System for Biological Resources

Contact Info

Product

Resources

About