Estimating Cardinalities with Deep Sketches

Kipf, Andreas; Vorona, Dimitri; Müller, Jonas; Kipf, Thomas; Radke, Bernhard; Leis, Viktor; Boncz, Peter; Neumann, Thomas; Kemper, Alfons

doi:10.48550/arxiv.1904.08223

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using machine learning techniques and deep neural networks is a recent trend in query optimization. Join order enumeration [34,27,33], cardinality estimation [32,31,24,25,57,42], selectivity estimation [61,15,12], and index structures [26] have been active research directions. Regarding the cardinality estimation problem, Malik et al [32] propose to train neural network models based on cardinality distributions for a separate class of similar queries and estimate overall query cardinalities.…”

Section: Related Workmentioning

confidence: 99%

Online Sketch-based Query Optimization

Izenov,

Datta,

Rusu

et al. 2021

Preprint

View full text Add to dashboard Cite

Cost-based query optimization remains a critical task in relational databases even after decades of research and industrial development. Query optimizers rely on a large range of statistical synopses -including attribute-level histograms and table-level samples -for accurate cardinality estimation. As the complexity of selection predicates and the number of join predicates increase, two problems arise. First, statistics cannot be incrementally composed to effectively estimate the cost of the sub-plans generated in plan enumeration. Second, small errors are propagated exponentially through join operators, which can lead to severely sub-optimal plans.In this paper, we introduce COMPASS, a novel query optimization paradigm for in-memory databases based on a single type of statistics-Fast-AGMS sketches. In COMPASS, query optimization and execution are intertwined. Selection predicates and sketch updates are pushed-down and evaluated online during query optimization. This allows Fast-AGMS sketches to be computed only over the relevant tuples-which enhances cardinality estimation accuracy. Plan enumeration is performed over the query join graph by incrementally composing attribute-level sketches-not by building a separate sketch for every sub-plan.We prototype COMPASS in MapD -an open-source parallel database -and perform extensive experiments over the complete JOB benchmark. The results prove that COMPASS generates better execution plans -both in terms of cardinality and runtime -compared to four other database systems. Overall, COMPASS achieves a speedup ranging from 1.35X to 11.28X in cumulative query execution time over the considered competitors.

show abstract

Section: Related Workmentioning

confidence: 99%

Online Sketch-based Query Optimization

Izenov,

Datta,

Rusu

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The implementation of this method is relatively simple and it is widely used, but it is ineffective when estimating the correlation between different columns. Data sketching [16][17][18][19] is also a summary-based cardinality estimation method, the core idea of which is to use a hash function to map a tuple value to a set of positions on a bitmap and to add count values to the corresponding positions according to the number of tuple values. The cardinality number of the tuple values can be inferred by counting the number of consecutive zeros or hits in the bitmap.…”

mentioning

confidence: 99%

A Cardinality Estimator in Complex Database Systems Based on TreeLSTM

Qi,

Yu,

2023

Sensors

View full text Add to dashboard Cite

Cardinality estimation is critical for database management systems (DBMSs) to execute query optimization tasks, which can guide the query optimizer in choosing the best execution plan. However, traditional cardinality estimation methods cannot provide accurate estimates because they cannot accurately capture the correlation between multiple tables. Several recent studies have revealed that learning-based cardinality estimation methods can address the shortcomings of traditional methods and provide more accurate estimates. However, the learning-based cardinality estimation methods still have large errors when an SQL query involves multiple tables or is very complex. To address this problem, we propose a sampling-based tree long short-term memory (TreeLSTM) neural network to model queries. The proposed model addresses the weakness of traditional methods when no sampled tuples match the predicates and considers the join relationship between multiple tables and the conjunction and disjunction operations between predicates. We construct subexpressions as trees using operator types between predicates and improve the performance and accuracy of cardinality estimation by capturing the join-crossing correlations between tables and the order dependencies between predicates. In addition, we construct a new loss function to overcome the drawback that Q-error cannot distinguish between large and small cardinalities. Extensive experimental results from real-world datasets show that our proposed model improves the estimation quality and outperforms traditional cardinality estimation methods and the other compared deep learning methods in three evaluation metrics: Q-error, MAE, and SMAPE.

show abstract

Estimating Cardinalities with Deep Sketches

Cited by 2 publications

References 0 publications

Online Sketch-based Query Optimization

Online Sketch-based Query Optimization

A Cardinality Estimator in Complex Database Systems Based on TreeLSTM

Contact Info

Product

Resources

About