Query Rewriting for Heterogeneous Data Lakes

Hai, Rihan; Quix, Christoph; Zhou, Chen

doi:10.1007/978-3-319-98398-1_3

Cited by 33 publications

(30 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, Spark SQL and SQL++ may be used to query both relational DBMSs and semi-structured data in JSON format. In addition, the Scalable Query Rewriting Engine (SQRE) handles graph databases [19]. Finally, CloudMdsQL also helps simultaneously query multiple relational and NoSQL DBMSs [30].…”

Section: Data Accessmentioning

confidence: 99%

On data lake architectures and metadata management

2020

View full text Add to dashboard Cite

Over the past two decades, we have witnessed an exponential increase of data production in the world. So-called big data generally come from transactional systems, and even more so from the Internet of Things and social media. They are mainly characterized by volume, velocity, variety and veracity issues. Big data-related issues strongly challenge traditional data management and analysis systems. The concept of data lake was introduced to address them. A data lake is a large, raw data repository that stores and manages all company data bearing any format. However, the data lake concept remains ambiguous or fuzzy for many researchers and practitioners, who often confuse it with the Hadoop technology. Thus, we provide in this paper a comprehensive state of the art of the different approaches to data lake design. We particularly focus on data lake architectures and metadata management, which are key issues in successful data lakes. We also discuss the pros and cons of data lakes and their design alternatives.

show abstract

Section: Data Accessmentioning

confidence: 99%

On data lake architectures and metadata management

2020

View full text Add to dashboard Cite

show abstract

“…The studies by Hai, Quix & Zhou (2018); Hricov et al (2017) andHsu, Liao &Shih (2012) present the results of their experiments processing XML files in terms of query execution time. However, the features of their Big Data ecosystem differ from ours and they do not present the versions used for the software.…”

Section: Comparison With Other Approachesmentioning

confidence: 99%

Efficient processing of complex XSD using Hive and Spark

Martinez-Mosquera

Navarrete

Luján‐Mora

2021

PeerJ Computer Science

View full text Add to dashboard Cite

The eXtensible Markup Language (XML) files are widely used by the industry due to their flexibility in representing numerous kinds of data. Multiple applications such as financial records, social networks, and mobile networks use complex XML schemas with nested types, contents, and/or extension bases on existing complex elements or large real-world files. A great number of these files are generated each day and this has influenced the development of Big Data tools for their parsing and reporting, such as Apache Hive and Apache Spark. For these reasons, multiple studies have proposed new techniques and evaluated the processing of XML files with Big Data systems. However, a more usual approach in such works involves the simplest XML schemas, even though, real data sets are composed of complex schemas. Therefore, to shed light on complex XML schema processing for real-life applications with Big Data tools, we present an approach that combines three techniques. This comprises three main methods for parsing XML files: cataloging, deserialization, and positional explode. For cataloging, the elements of the XML schema are mapped into root, arrays, structures, values, and attributes. Based on these elements, the deserialization and positional explode are straightforwardly implemented. To demonstrate the validity of our proposal, we develop a case study by implementing a test environment to illustrate the methods using real data sets provided from performance management of two mobile network vendors. Our main results state the validity of the proposed method for different versions of Apache Hive and Apache Spark, obtain the query execution times for Apache Hive internal and external tables and Apache Spark data frames, and compare the query performance in Apache Hive with that of Apache Spark. Another contribution made is a case study in which a novel solution is proposed for data analysis in the performance management systems of mobile networks.

show abstract

“…Constance 84,85 is a Data Lake (DL) system with sophisticated metadata management over raw data extracted from heterogeneous data sources. Constance discovers, extracts, and summarizes the structural metadata from the data sources, and annotates data and metadata with semantic information to avoid ambiguities.…”

Section: Related Workmentioning

confidence: 99%

“…The maintenance layer, which dynamically and incrementally extracts and summarizes the current metadata of the DL, provides a uniform query interface. Recently, the authors 85 tackle the challenge of query rewriting in Constance and propose an extensible framework of the query rewriting engine in their data lake system to resolve the schematic heterogeneity of how the data is structured, and the heterogeneity of underlying data stores. They have designed and implemented a set of novel approaches that aim to query both relational and NoSQL data stores without the overhead of tremendous data transformation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

IPDS: A semantic mediator‐based system using Spark for the integration of heterogeneous proteomics data sources

Messaoudi

Fissoune

Badir

2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary With the constant rise of data volumes in many disciplines, various new Big data management systems have emerged to provide scalable tools for efficient data integration, processing, and analysis. In this article, we provide an overview of biomedical data integration systems focusing on ontology‐based semantic systems and Big data technologies based systems such as Apache Spark. We also propose a new semantic data integration system, called Integrated Proteomics Data System (IPDS), which uses a mediator approach. IPDS provides users a unified interface for query processing and data exploration. This system takes advantage of the Apache Spark framework to perform the query transformation and execution needed to question the integrated data sources. We develop a domain ontology that allows the user to formulate its queries in terms defined in the ontology. IPDS is a case study of semantic proteomics data integration linking four data sources UniProt (protein annotation), String (protein‐protein interaction), PDB (protein structure), and Pubmed (biomedical citation).

show abstract

Query Rewriting for Heterogeneous Data Lakes

Cited by 33 publications

References 11 publications

On data lake architectures and metadata management

On data lake architectures and metadata management

Efficient processing of complex XSD using Hive and Spark

IPDS: A semantic mediator‐based system using Spark for the integration of heterogeneous proteomics data sources

Contact Info

Product

Resources

About