Data lake concept and systems: a survey

Abstract

Although big data has been discussed for some years, it still has many research challenges, especially the variety of data. It poses a huge difficulty to efficiently integrate, access, and query the large volume of diverse data in information silos with the traditional 'schema-on-write' approaches such as data warehouses. Data lakes have been proposed as a solution to this problem. They are repositories storing raw data in its original formats and providing a common access interface. This survey reviews the development, definition, and architectures of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing data lake systems based on their provided functions, which makes this survey a useful technical reference for designing, implementing and applying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey would motivate the future development of data lake research and practice.

CEBA: A Data Lake for Data Sharing and Environmental Monitoring

“…Finally, Hai et al [ 54 ] propose a new definition: “A data lake is a flexible and scalable data storage and management system that ingests and stores raw data from heterogeneous sources in its original format, and provides query processing and data analysis on the fly.” A data lake is not considered only as a storage system and must support on-demand data processing and querying. Furthermore, data indexing should only be performed if necessary at the time of data access, ingestion of data sources could be light, as there is no need to force schema definitions and mappings beforehand.…”

“…Hai et al [ 54 ] identify three examples of real applications of data lakes. Our project is of the type “multiple entries of heterogeneous raw data” with additions on: Managing geographic coordinates; Having a full JSON architecture for data querying; Considering a data valid for the infrastructure as soon as it has a “what, where, when” information; Combining relational and not only SQL (NoSQL) storage and querying systems; Giving the access to the data to any kind of users, including open data users; Furthermore, finally managing metadata with a catalogue managing spatially referenced resources.…”

Toward data lakes as central building blocks for data management and analysis

“…As of today, a lot of development and analysis was conducted in the area of data lake architectures , where the so-called zone architecture (Patel et al, 2017 ; Ravat and Zhao, 2019 ), including the pond architecture (Inmon, 2016 ), became the most cited and used. These architectures have already been surveyed by Hai et al ( 2021 ) and Sawadogo and Darmont ( 2021 ), and a functional architecture was proposed by both of them, and a maturity and a hybrid architecture have been derived by Sawadogo and Darmont ( 2021 ). These surveys, however, did not include recent works like the definition of a zone reference model (Giebler et al, 2020 ) or a data lake architecture based on FAIR Digital Objects (FDOs) (Nolte and Wieder, 2022 ).…”

“…Within a functional -based architecture classification, the data lake is analyzed toward its operations which are performed on the data while moving through the general data lake workflow. Hai et al ( 2021 ) define three layers, ingestion, maintenance , and exploration , where corresponding functions are then sub-grouped. A similar definition is provided by Sawadogo and Darmont ( 2021 ), where the four main components of a data lake are defined as ingestion, storage, processing, querying .…”

Processing Analytical Queries over Polystore System for a Large Astronomy Data Repository

“…There is a strong emphasis on data science and analysis. Data science is frequently performed on vast repositories, sometimes referred to as data lakes, that contain a large number of distinct datasets [8]. The datasets may be sparsely or completely schema-less [9].…”