Rule-based multi-query optimization

Abstract

Data stream management systems usually have to process many long-running queries that are active at the same time. Multiple queries can be evaluated more efficiently together than independently, because it is often possible to share state and computation. Motivated by this observation, various Multi-Query Optimization (MQO) techniques have been proposed. However, these approaches suffer from two limitations. First, they focus on very specialized workloads. Second, integrating MQO techniques for CQL-style stream engines and those for event pattern detection engines is even harder, as the processing models of these two types of stream engines are radically different.In this paper, we propose a rule-based MQO framework. This framework incorporates a set of new abstractions, extending their counterparts, physical operators, transformation rules, and streams, in a traditional RDBMS or stream processing system. Within this framework, we can integrate new and existing MQO techniques through the use of transformation rules. This allows us to build an expressive and scalable stream system. Just as relational optimizers are crucial for the success of RDBMSes, a powerful multi-query optimizer is needed for data stream processing. This work lays the foundation for such a multi-query optimizer, creating opportunities for future research. We experimentally demonstrate the efficacy of our approach.

Joint Top-K Spatial Keyword Query Processing

“…Techniques for the efficient processing of one group of queries [8], [10], [18], [22]. For example, sub-expressions shared among a set of SQL queries can be evaluated once and then subsequently reused.…”

“…Specifically, the key of e is defined as its minimum distance to its relevant subqueries of Q. The loop has two phases: (I) checking whether the dequeued entry e can contribute a closer and relevant result for some subquery (lines [8][9][10][11][12][13][14][15][16][17], and (II) processing the child node of e (lines [18][19][20][21][22][23][24][25][26][27][28][29][30]. Algorithm 2 GROUP (Joint query Q, Tree root root, Integer k)…”

“…If |D| < B/2 then D is returned to the parent algorithm call (lines 3-4) for further processing. If |D| > B (line 5) then we partition D (lines [6][7][8][9][10][11][12][13][14][15][16][17][18][19].…”

“…With a typical node capacity in the hundreds and a fill-factor of approximately 0.7, the leaf level makes up well beyond 99% of the index. 8 be any non-leaf entry that points to a leaf node. We need to access e's leaf node when: 1) the keyword set of e contains q.ψ, and 2) the minimum distance from q to e is within d knn .…”

CEPSim: A Simulator for Cloud-Based Complex Event Processing

“…Finally, some systems are based on pattern-based query languages that cannot be directly mapped to DAGs. Even in this case, however, previous studies [17] have shown that is possible to transform these queries into DAG structures.…”