Selectivity Estimation on Set Containment Search

Abstract

In this paper, we study the problem of selectivity estimation on set containment search. Given a query record Q and a record dataset S , we aim to accurately and efficiently estimate the selectivity of set containment search of query Q over S. We first extend existing distinct value estimating techniques to solve this problem and develop an inverted list and G-KMV sketchbased approach IL-GKMV. We analyze that the performance of IL-GKMV degrades with the increase in vocabulary size. Motivated by limitations of existing techniques and the inherent challenges of the problem, we resort to developing effective and efficient sampling approaches and propose an ordered trie structure-based sampling approach named OT-Sampling. OT-Sampling partitions records based on element frequency and occurrence patterns and is significantly more accurate compared with simple random sampling method and IL-GKMV. To further enhance the performance, a divide-and-conquer-based sampling approach, DC-Sampling, is presented with an inclusion/exclusion prefix to explore the pruning opportunities. Meanwhile, we consider weighted set containment selectivity estimation and devise stratified random sampling approach named StrRS. We theoretically analyze the proposed techniques regarding various accuracy estimators. Our comprehensive experiments on nine real datasets verify the effectiveness and efficiency of our proposed techniques.

A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration

“…Figure 1 illustrates the three most important components in a cost-based optimizer: cardinality estimation (CE), cost model (CM), and plan enumeration (PE). CE uses statistics of data and some assumptions about data distribution, column correlation, and join relationship to get the number of tuples generated by an intermediate operator, 1 which is also crucial for other search problems, e.g., [101,102]. CM can be regarded as a complex function that maps the current state of database and estimated cardinalities to the cost of executing a (sub)plan.…”