Improved Cardinality Estimation by Learning Queries Containment Rates

Abstract

The containment rate of query Q1 in query Q2 over database D is the percentage of Q1's result tuples over D that are also in Q2's result over D. We directly estimate containment rates between pairs of queries over a specific database. For this, we use a specialized deep learning scheme, CRN, which is tailored to representing pairs of SQL queries. Resultcardinality estimation is a core component of query optimization. We describe a novel approach for estimating queries result-cardinalities using estimated containment rates among queries. This containment rate estimation may rely on CRN or embed, unchanged, known cardinality estimation methods. Experimentally, our novel approach for estimating cardinalities, using containment rates between queries, on a challenging real-world database, realizes significant improvements to state of the art cardinality estimation methods.

Learning From User-Specified Optimizer Hints in Database Systems

“…Machine Learning has been recently considered a promising technique for database query optimization, e.g. Leo [24], which adjusted histogram estimators by monitoring similar queries, [10][15][25] [28] proposed to use deep learning to learn cardinality estimations or query costs, in [1][2][3] cardinality estimation is based on query driven approaches, [29] [30] described unsupervised Monte Carlo-based solutions, in [8] cardinalities were estimated via query containment rates, [12][17] demonstrated that reinforcement learning helps find good query execution plans. In [10] the authors proposed to use a multi-set convolutional network to predict join-crossing correlations in the data.…”

Towards instance-optimized data systems

“…Unsupervised approaches, based on Monte Carlo integration, have also been proposed [116,117]. In [43], the authors present a scheme called CRN for estimating cardinalities via query containment rates.…”