AQUA+: Query Optimization for Hybrid Database-MapReduce System

Pang, Zhifei; Wu, Sai; Huang, Haichao; Hong, Zhouzhenyan; Xie, Yuqing

doi:10.1007/s10115-020-01542-4

Cited by 3 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This section provides the discussion about proposed method and its result comparisons. The AQUA+ [25] has limitation such as due to an insufficient map-reduce engine, the sharding and filtering performance gets worse when the cluster gets larger. The existing model JQPro [30] has limitations such as the semantic data capacity is maximized with the amount of RDF datasets excessive of trillion triples in RDF repositories.…”

Section: Discussionmentioning

confidence: 99%

“…The section shows the comparative analysis of proposed method with existing methods like AQUA+ [25] and join query processing (JQPro) [30] in terms of execution time. Two alternative cases are examined while analyzing the proposed method.…”

Section: Comparative Analysismentioning

confidence: 99%

“…In case 1, the execution time of query is taken in seconds where as in case 2, the execution time of query is taken in milliseconds. Table 3, the proposed method is compared to AQUA+ [25] for  ISSN: 2302-9285 case 1. Table 4, shows the comparative analysis of proposed method with JQPro [30] for case.…”

Section: Comparative Analysismentioning

confidence: 99%

“…The MF-TSA query optimizer used integrating tables for selecting the cost of join and it was processed only once. Pang et al [25] implemented an hybrid database-map reduction system (AQUA+) model for query optimization in a hybrid database-map reduction system. In this model, the distributed database was used for maintaining the data, and on top of the same set of cluster nodes map reduce engines were deployed.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Structured query language query join optimization by using rademacher averages and mapreduce algorithms

Chandrashekariah,

H. A.

2024

Bulletin EEI

View full text Add to dashboard Cite

Query optimization involves identifying and implementing the most effective and efficient methods and strategies to enhance the performance of queries. This is achieved by intelligently utilizing system resources and considering various performance metrics. Table joining optimization involves optimizing the process of combining two or more tables within a database. Structured query language (SQL) optimization is the progress of utilizing SQL queries in the possible way to achieve fast and accurate database results. SQL optimization is critical to decreasing the no of queries in research description framework (RDF) and the time for processing a huge number of relatable data. In this paper, four new algorithms are proposed such as hash-join, sort-merge, rademacher averages and mapreduce for the progress of SQL query join optimization. The proposed model is evaluated and tested using waterloo sparql diversity test suite (WatDiv) and lehigh university benchmark (LUBM) benchmark datasets in terms of time execution. The results represented that the proposed method achieved an enhanced performance of less execution time for various queries such as Q3 of 5362, Q8 of 5921, Q9 of 5854 and Q10 of 5691 milliseconds. The proposed gives better performance than other existing methods like hybrid database-map reduction system (AQUA+) and join query processing (JQPro).

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Comparative Analysismentioning

confidence: 99%

Section: Comparative Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Structured query language query join optimization by using rademacher averages and mapreduce algorithms

Chandrashekariah,

H. A.

2024

Bulletin EEI

View full text Add to dashboard Cite

show abstract

“…On the other hand, as a popular programming model, MapReduce has also been leveraged to facilitate query processing and used as a database engine [28,21,42,43,44,45]. Several translators, including Pig Latin/Pig [26], SCOPE [27], HiveQL/Hive [28], YSmart [21], and Gerenuk [46], have been developed to allow the queries expressed in SQL-like query languages to be translated into MapReduce scripts automatically.…”

Section: Related Workmentioning

confidence: 99%

A Novel Approach to Translate Structural Aggregation Queries to MapReduce Code

Abdelmoniem

Atwa

Abdulah

2022

Preprint

View full text Add to dashboard Cite

Data management applications are rapidly growing applications that require more attention, especially in the big data era. Thus, it is critical to support these applications with novel and efficient algorithms that satisfy higher performance. Array database management systems are one way to support these applications by dealing with data represented in n-dimensional data structures. For instance, software like SciDB and RasDaMan can be powerful tools to achieve the required performance on large-scale problems with multidimensional data. Like their relational counterparts, these management systems support specific array query languages as the user interface.Further, as a popular programming model, MapReduce allows large-scale data analysis and has also been leveraged to facilitate query processing and used as a database engine. Nevertheless, one major obstacle is the low productivity of developing MapReduce applications. Unlike the high-level declarative language such as SQL, MapReduce jobs are written in low-level descriptive language, often requiring massive programming efforts and complicated programming debugging processes.This paper presents a system that supports translating array queries expressed by AQL (Array Query Language) in SciDB into MapReduce jobs. We focus on effectively translating some unique structural aggregations, including circular, grid, hierarchical, and sliding aggregations. Unlike the traditional aggregations in relational databases, these structural aggregations are designed explicitly for array manipulation. Thus, our work can be considered an array-view counterpart of some existing SQL-to- MapReduce translators like HiveQL/Hive and YSmart. We show that our translator can effectively support structural aggregations over arrays (or sub-arrays) to meet various array manipulations. Moreover, our translator can help user-defined aggregation functions with the minimum effort of the user.We also show that our translator can generate optimized MapReduce code, leading to significantly better performance than the short hand-written code by up to 10.84X.

show abstract

AutoSteer: Learned Query Optimization for Any SQL Database

Anneser,

Tatbul,

Cohen

et al. 2023

Proc. VLDB Endow.

View full text Add to dashboard Cite

This paper presents AutoSteer, a learning-based solution that automatically drives query optimization in any SQL database that exposes tunable optimizer knobs. AutoSteer builds on the Bandit optimizer (Bao) and extends it with new capabilities (e.g., automated hint-set discovery) to minimize integration effort and facilitate usability in both monolithic and disaggregated SQL systems. We successfully applied AutoSteer on PostgreSQL, PrestoDB, Spark-SQL, MySQL, and DuckDB - five popular open-source database engines with diverse query optimizers. We then conducted a detailed experimental evaluation with public benchmarks (JOB, Stackoverflow, TPC-DS) and a production workload from Meta's PrestoDB deployments. Our evaluation shows that AutoSteer can not only outperform these engines' native query optimizers (e.g., up to 40% improvements for PrestoDB) but can also match the performance of Bao-for-PostgreSQL with reduced human supervision and increased adaptivity, as it replaces Bao's static, expert-picked hint-sets with those that are automatically discovered. We also provide an open-source implementation of AutoSteer together with a visual tool for interactive use by query optimization experts.

show abstract

AQUA+: Query Optimization for Hybrid Database-MapReduce System

Cited by 3 publications

References 18 publications

Structured query language query join optimization by using rademacher averages and mapreduce algorithms

Structured query language query join optimization by using rademacher averages and mapreduce algorithms

A Novel Approach to Translate Structural Aggregation Queries to MapReduce Code

AutoSteer: Learned Query Optimization for Any SQL Database

Contact Info

Product

Resources

About