Efficient answering of set containment queries for skewed item distributions

Terrovitis, Manolis; Bouros, Panagiotis; Vassiliadis, Panos; Sellis, Timos; Mamoulis, Nikos

doi:10.1145/1951365.1951394

Cited by 22 publications

(20 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(1) Our empirical study showed that skewed data is challenging for our algorithms. Incorporation in our algorithms of recent results on efficiently dealing with list intersections and data skew should be investigated [5,35]. (2) It would be interesting to study the impact on our solutions of variations to the data model (e.g., multi-set and list types) or query model (e.g., bottom-up subtree embeddings [36]).…”

Section: Discussionmentioning

confidence: 99%

“…In the literature on processing containment queries on flat sets, solutions using inverted files as the physical representation of the database have demonstrated robust efficient performance [15,24,35]. 4 Furthermore, a variety of industrial-strength open-source solutions for building inverted files are available off the shelf, and are widely adopted and used by practitioners.…”

Section: Preliminariesmentioning

confidence: 99%

“…Consequently, efficient processing of set containment queries has been heavily investigated. Basic approaches include the use of inverted files, signature files, and hashing to facilitate efficient join processing (e.g., [15,24,25,28,35,34]). To the best of our knowledge, however, all known solutions for processing containment queries are restricted to the special case of flat sets, i.e., sets without nesting, whose members are all atomic objects.…”

Section: Introductionmentioning

confidence: 99%

“…Although we build in our work on the same mature inverted file data structure used by TPQ solutions for representing XML trees (e.g., the classic position-based node encoding used in [8]), the richer structure of nested sets necessarily requires a fresh re-think of query processing solutions. Indeed, it has already been observed that state-of-the-art TPQ solutions are inadequate even in the special case of flat set containment joins [35]. Finally, solutions for nested relational join processing are designed and optimized for the restricted special case of nested sets having fixed pre-defined input schemas, and hence are not suitable for processing collections of heterogeneous nested sets [9].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Efficient processing of containment queries on nested sets

Ibrahim

Fletcher

2013

Proceedings of the 16th International Conference on Extending Database Technology

View full text Add to dashboard Cite

We study the problem of computing containment queries on sets which can have both atomic and set-valued objects as elements, i.e., nested sets. Containment is a fundamental query pattern with many basic applications. Our study of nested set containment is motivated by the ubiquity of nested data in practice, e.g., in XML and JSON data management, in business and scientific workflow management, and in web analytics. Furthermore, there are to our knowledge no known efficient solutions to computing containment queries on massive collections of nested sets. Our specific contributions in this paper are: (1) we introduce two novel algorithms for efficient evaluation of containment queries on massive collections of nested sets; (2) we study caching and filtering mechanisms to accelerate query processing in the algorithms; (3) we develop extensions to the algorithms to a) compute several related query types and b) accommodate natural variations of the semantics of containment; and, (4) we present analytic and empirical analyses which demonstrate that both algorithms are efficient and scalable.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient processing of containment queries on nested sets

Ibrahim

Fletcher

2013

Proceedings of the 16th International Conference on Extending Database Technology

View full text Add to dashboard Cite

show abstract

“…Experimental studies [22,44] showed that inverted files outperform signature-based indices for set containment queries on datasets with low cardinality set objects, e.g., typical text databases. In [37,38], the authors proposed extensions of the classic inverted file data structure, which optimize the indexing set-valued data with skewed item distribu-tions. In [14], the authors proposed an indexing scheme for text documents, which includes inverted lists for frequent word combinations.…”

Section: Set Containment Queriesmentioning

confidence: 99%

Set containment join revisited

Bouros

Mamoulis

et al. 2015

Knowl Inf Syst

Self Cite

View full text Add to dashboard Cite

Given two collections of set objects R and S, the R ⊆ S set containment join returns all object pairs (r, s) ∈ R × S such that r ⊆ s. Besides being a basic operator in all modern data management systems with a wide range of applications, the join can be used to evaluate complex SQL queries based on relational division and as a module of data mining algorithms. The state-of-the-art algorithm for set containment joins (PRETTI) builds an inverted index on the right-hand collection S and a prefix tree on the left-hand collection R that groups set objects with common prefixes and thus, avoids redundant processing. In this paper, we present a framework which improves PRETTI in two directions. First, we limit the prefix tree construction by proposing an adaptive methodology based on a cost model; this way, we can greatly reduce the space and time cost of the join. Second, we partition the objects of each collection based on their first contained item, assuming that the set objects are internally sorted. We show that we can process the partitions and evaluate the join while building the prefix tree and the inverted index progressively. This allows us to significantly reduce not only the join cost, but also the maximum memory requirements during the join.

show abstract

Selectivity Estimation on Set Containment Search

Yang

Zhang

et al. 2019

Database Systems for Advanced Applications

View full text Add to dashboard Cite

In this paper, we study the problem of selectivity estimation on set containment search. Given a query record Q and a record dataset S, we aim to accurately and efficiently estimate the selectivity of set containment search of query Q over S. The problem has many important applications in commercial fields and scientific studies. To the best of our knowledge, this is the first work to study this important problem. We first extend existing distinct value estimating techniques to solve this problem and develop an inverted list and G-KMV sketch based approach IL-GKMV. We analyse that the performance of IL-GKMV degrades with the increase of vocabulary size. Motivated by limitations of existing techniques and the inherent challenges of the problem, we resort to developing effective and efficient sampling approaches and propose an ordered trie structure based sampling approach named OT-Sampling. OT-Sampling partitions records based on element frequency and occurrence patterns and is significantly more accurate compared with simple random sampling method and IL-GKMV. To further enhance performance, a divide-and-conquer based sampling approach, DC-Sampling, is presented with an inclusion/exclusion prefix to explore the pruning opportunities. We theoretically analyse the proposed techniques regarding various accuracy estimators. Our comprehensive experiments on 6 real datasets verify the effectiveness and efficiency of our proposed techniques.

show abstract

Efficient answering of set containment queries for skewed item distributions

Cited by 22 publications

References 46 publications

Efficient processing of containment queries on nested sets

Efficient processing of containment queries on nested sets

Set containment join revisited

Selectivity Estimation on Set Containment Search

Contact Info

Product

Resources

About