Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload

Kang, Johan Kok Zhi; Gaurav, .; Tan, Sien Yi; Cheng, Feng; Sun, Shixuan; He, Bingsheng

doi:10.1145/3448016.3457546

Cited by 18 publications

(6 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, it is becoming a common practice to build largescale commercial data lakes on cloud infrastructure [144], [148], [152]. Cloud-based storage choices include singlecloud, multi-cloud and a hybrid of cloud and on-premise platforms [148].…”

Section: Cloud Data Lakesmentioning

confidence: 99%

Data Lakes: A Survey of Functions and Systems

Hai

Koutras

Quix

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Data lakes are becoming increasingly prevalent for big data management and data analytics. In contrast to traditional 'schema-on-write' approaches such as data warehouses, data lakes are repositories storing raw data in its original formats and providing a common access interface. Despite the strong interest raised from both academia and industry, there is a large body of ambiguity regarding the definition, functions and available technologies for data lakes. A complete, coherent picture of data lake challenges and solutions is still missing. This survey reviews the development, architectures, and systems of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing approaches and systems based on their provided functions for data lakes, which makes this survey a useful technical reference for designing, implementing and deploying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey will motivate the future development of data lake research and practice.

show abstract

Section: Cloud Data Lakesmentioning

confidence: 99%

Data Lakes: A Survey of Functions and Systems

Hai

Koutras

Quix

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…Others focus on refining traditional cost models and plan enumeration algorithms. For learned cost models, [41] and [55,57] utilize TreeL-STM and convolution models to learn cost of single and concurrent queries, respectively. Plan enumeration is often modelled as a reinforcement learning problem on deciding the best join order of tables.…”

Section: Related Workmentioning

confidence: 99%

Lero: A Learning-to-Rank Query Optimizer

Zhu¹,

Chen²,

Ding³

et al. 2023

Preprint

View full text Add to dashboard Cite

A recent line of works apply machine learning techniques to assist or rebuild cost-based query optimizers in DBMS. While exhibiting superiority in some benchmarks, their deficiencies, e.g., unstable performance, high training cost, and slow model updating, stem from the inherent hardness of predicting the cost or latency of execution plans using machine learning models. In this paper, we introduce a learning-to-rank query optimizer, called Lero, which builds on top of a native query optimizer and continuously learns to improve the optimization performance. The key observation is that the relative order or rank of plans, rather than the exact cost or latency, is sufficient for query optimization. Lero employs a pairwise approach to train a classifier to compare any two plans and tell which one is better. Such a binary classification task is much easier than the regression task to predict the cost or latency, in terms of model efficiency and accuracy. Rather than building a learned optimizer from scratch, Lero is designed to leverage decades of wisdom of databases and improve the native query optimizer. With its non-intrusive design, Lero can be implemented on top of any existing DBMS with minimal integration efforts. We implement Lero and demonstrate its outstanding performance using PostgreSQL. In our experiments, Lero achieves near optimal performance on several benchmarks. It reduces the plan execution time of the native optimizer in PostgreSQL by up to 70% and other learned query optimizers by up to 37%. Meanwhile, Lero continuously learns and automatically adapts to query workloads and changes in data.

show abstract

“…Cardinality/selectivity estimation, has improved considerably leveraging ML [17,70,77,78,84]. Likewise for query optimization [27,44,45], indexes [9,10,30,49], cost estimation [63,83], workload forecasting [85], DB tuning [34,68,81], synthetic data generation [7,54,76], etc.…”

Section: Introductionmentioning

confidence: 99%

Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data

Kurmanji,

Triantafillou

2023

Proc. ACM Manag. Data

View full text Add to dashboard Cite

Machine Learning (ML) is changing DBs as many DB components are being replaced by ML models. One open problem in this setting is how to update such ML models in the presence of data updates. We start this investigation focusing on data insertions (dominating updates in analytical DBs). We study how to update neural network (NN) models when new data follows a different distribution (a.k.a. it is "out-of-distribution" -- OOD), rendering previously-trained NNs inaccurate. A requirement in our problem setting is that learned DB components should ensure high accuracy for tasks on old and new data (e.g., for approximate query processing (AQP), cardinality estimation (CE), synthetic data generation (DG), etc.). This paper proposes a novel updatability framework (DDUp). DDUp can provide updatability for different learned DB system components, even based on different NNs, without the high costs to retrain the NNs from scratch. DDUp entails two components: First, a novel, efficient, and principled statistical-testing approach to detect OOD data. Second, a novel model updating approach, grounded on the principles of transfer learning with knowledge distillation, to update learned models efficiently, while still ensuring high accuracy. We develop and showcase DDUp's applicability for three different learned DB components, AQP, CE, and DG, each employing a different type of NN. Detailed experimental evaluation using real and benchmark datasets for AQP, CE, and DG detail DDUp's performance advantages.

show abstract

Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload

Cited by 18 publications

References 32 publications

Data Lakes: A Survey of Functions and Systems

Data Lakes: A Survey of Functions and Systems

Lero: A Learning-to-Rank Query Optimizer

Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data

Contact Info

Product

Resources

About