Data Lake Architecture

Abstract

Data is the biggest asset after people for businesses, and it is a new driver of the world economy. The volume of data that enterprises gather every day is growing rapidly. This kind of rapid growth of data in terms of volume, variety, and velocity is known as Big Data. Big Data is a challenge for enterprises, and the biggest challenge is how to store Big Data. In the past and some organizations currently, data warehouses are used to store Big Data. Enterprise data warehouses work on the concept of schema-on-write but Big Data analytics want data storage which works on the schema-on-read concept. To fulfill market demand, researchers are working on a new data repository system for Big Data storage known as a data lake. The data lake is defined as a data landing area for raw data from many sources. There is some confusion and questions which must be answered about data lakes. The objective of this article is to reduce the confusion and address some question about data lakes with the help of architecture.

Construction and Application of a Big Data System for Regional Lakes in Coalbed Methane Development

“…Many companies have proposed different data integration solutions for it. Huawei Cloud data lake solution is based on the advanced cloud system architecture and provides the data lake Query Analysis Service (DLI) . As a Spark-based data integration and exploration service of the Huawei Cloud data lake, DLI solves the following difficulties when users use data: (a) Traditional use of big data requires tedious steps and has certain requirements for the objective environment.…”

“…Huawei Cloud data lake solution is based on the advanced cloud system architecture and provides the data lake Query Analysis Service (DLI). 8 As a Spark-based data integration and exploration service of the Huawei Cloud data lake, DLI solves the following difficulties when users use data: (a) Traditional use of big data requires tedious steps and has certain requirements for the objective environment. This includes the configuration of computer rooms, networks, clusters, etc., followed by the installation and application of relevant system platforms.…”

Intelligent Decision-Making System for Integrated Geological and Engineering of Deep Coalbed Methane Development

“…12 The fracturing fluid system still cannot meet the requirements of low damage and fouling transformation, and further technology development is needed. 13 Facing the new unconventional gas reservoir type of deep coalbed methane, there is an urgent need to solve numerous basic theories and technical bottlenecks. 14 This is the only way to achieve gas storage and production increase and effective economic development.…”

“…There is still considerable room for improvement in the effective fracture network volume . The fracturing fluid system still cannot meet the requirements of low damage and fouling transformation, and further technology development is needed . Facing the new unconventional gas reservoir type of deep coalbed methane, there is an urgent need to solve numerous basic theories and technical bottlenecks .…”