Deep Learning Models for Selectivity Estimation of Multi-Attribute Queries

Hasan, Shohedul; Thirumuruganathan, Saravanan; Augustine, Jees; Koudas, Nick; Das, Gautam

doi:10.1145/3318464.3389741

Cited by 79 publications

(58 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hasan et al [35] also adopt the deep autoregressive models and introduce an adaptive sampling method to support range queries. Compared with the Naru, the authors adopt the binary encoding method and the sampling process runs parallelly, which leads the model is smaller than Naru and makes the inference faster.…”

Section: Unsupervised Methodsmentioning

confidence: 99%

A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration

2021

View full text Add to dashboard Cite

Query optimizer is at the heart of the database systems. Cost-based optimizer studied in this paper is adopted in almost all current database systems. A cost-based optimizer introduces a plan enumeration algorithm to find a (sub)plan, and then uses a cost model to obtain the cost of that plan, and selects the plan with the lowest cost. In the cost model, cardinality, the number of tuples through an operator, plays a crucial role. Due to the inaccuracy in cardinality estimation, errors in cost model, and the huge plan space, the optimizer cannot find the optimal execution plan for a complex query in a reasonable time. In this paper, we first deeply study the causes behind the limitations above. Next, we review the techniques used to improve the quality of the three key components in the cost-based optimizer, cardinality estimation, cost model, and plan enumeration. We also provide our insights on the future directions for each of the above aspects.

show abstract

Section: Unsupervised Methodsmentioning

confidence: 99%

A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration

2021

View full text Add to dashboard Cite

show abstract

“…Note that we use left-to-right order in this work, which was demonstrated to be effective in previous work [78]. More strategies for choosing a good ordering can be found in [28,78].…”

Section: Preliminary: Deep Autoregressive Models For Cardinality Estimationmentioning

confidence: 97%

“…Kernel density estimation (KDE)-based methods [25,26] do not need the independence assumptions, but their accuracy is not very competitive due to the difficulty in adjusting the bandwidth parameter. Recently, Naru [78] and MADE [28] utilize unsupervised deep autoregressive models for learning the conditional probability distribution and use it for answering point queries. Naru uses progressive sampling and MADE uses adaptive importance sampling algorithm for answering range queries and they achieve comparative results.…”

Section: Related Workmentioning

confidence: 99%

“…However, it will not handle well strong attribute correlations in real-world datasets. A promising recent advance in this direction would be applying deep autoregressive models [20,53,63]) for cardinality estimation [28,78], which can capture attribute correlations and have reasonable estimation efficiency. However, the optimization target (loss function) of deep autoregressive models is to minimize the ordinary average error over the overall data, and could neglect tricky (e.g., long tail) data regions, because the model may be largely dominated by those few head data regions, but degraded for many other tail data regions.…”

Section: Introductionmentioning

confidence: 99%

“…Our Solution To achieve the four goals, in this paper we propose a new unified deep autoregressive estimator UAE to utilize data as unsupervised information and query workload as supervised information for learning the joint data distribution. Deep autoregressive models have demonstrated superior performance for their effectiveness and efficiency in the pioneering work [28,78] of training autoregressive models for cardinality estimation. However, no existing deep autoregressive model in the literature is able to incorporate the query workload as supervised information for learning joint data distribution, much less support both data as unsupervised information and query workload as supervised information in the same model.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation

Wu¹,

Cong

2021

Proceedings of the 2021 International Conference on Management of Data

View full text Add to dashboard Cite

Cardinality estimation is a fundamental problem in database systems. To capture the rich joint data distributions of a relational table, most of the existing work either uses data as unsupervised information or uses query workload as supervised information. Very little work has been done to use both types of information, and cannot fully make use of both types of information to learn the joint data distribution. In this work, we aim to close the gap between data-driven and query-driven methods by proposing a new unified deep autoregressive model, UAE, that learns the joint data distribution from both the data and query workload. First, to enable using the supervised query information in the deep autoregressive model, we develop differentiable progressive sampling using the Gumbel-Softmax trick. Second, UAE is able to utilize both types of information to learn the joint data distribution in a single model. Comprehensive experimental results demonstrate that UAE achieves single-digit multiplicative error at tail, better accuracies over state-of-the-art methods, and is both space and time efficient.

show abstract

Query cost estimation in graph databases via emphasizing query dependencies by using a neural reasoning network

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryWith the increasing complexity of graph queries, query cost estimation has become a key challenge in graph databases. Accurate estimation results are critical for database administrators or database management systems to perform query processing or optimization tasks. An efficient and accurate estimation model can improve the estimation quality and make the produced results credible. Although learning‐based methods have been applied in query cost estimation, most of them are directed at relational queries and cannot be directly used for graph queries. Furthermore, most estimation approaches focus on the correlations between predicates or columns. The dependencies between query schema and query filter conditions and the correlation between query schema are ignored. In this study, we construct a novel deep learning model composed of reasoning and retrieval processes that can accurately capture the potential logical relationships in graph queries. This solves the above problems to some extent. In addition, we propose a query estimation framework that divides the estimation task into query workload generation, training data collection, feature extraction and encoding, and estimation model construction. The results of the experiment on real‐world datasets show that our estimation model can improve the estimation quality and outperforms other compared deep learning models in terms of estimation accuracy.

show abstract

Deep Learning Models for Selectivity Estimation of Multi-Attribute Queries

Cited by 79 publications

References 40 publications

A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration

A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration

A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation

Query cost estimation in graph databases via emphasizing query dependencies by using a neural reasoning network

Contact Info

Product

Resources

About