Adaptive Query Processing

Deshpande, Amol; Ives, Zachary G.; Raman, V.

doi:10.1561/9781601980359

Cited by 51 publications

(6 citation statements)

References 119 publications

(216 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Put simply, a small number of optimization mistakes leads to workload performance far below what is possible. Ideas to improve the optimizer in these cases include better selectivity estimates [2]- [4] providing more exotic search mechanisms for plans [5], [6], and changing query plans that have gone off the rails at runtime [7]- [11]. A more recent trend is to recast query optimization as a machine learning problem.…”

Section: Introductionmentioning

confidence: 99%

How I Learned to Stop Worrying and Love Re-optimization

Perron¹,

Shang²,

Kraska³

2019

2019 IEEE 35th International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Cost-based query optimizers remain one of the most important components of database management systems for analytic workloads. Though modern optimizers select plans close to optimal performance in the common case, a small number of queries are an order of magnitude slower than they could be. In this paper we investigate why this is still the case, despite decades of improvements to cost models, plan enumeration, and cardinality estimation. We demonstrate why we believe that a re-optimization mechanism is likely the most cost-effective way to improve end-to-end query performance. We find that even a simple re-optimization scheme can improve the latency of many poorly performing queries. We demonstrate that re-optimization improves the end-to-end latency of the top 20 longest running queries in the Join Order Benchmark by 27%, realizing most of the benefit of perfect cardinality estimation.

show abstract

Section: Introductionmentioning

confidence: 99%

How I Learned to Stop Worrying and Love Re-optimization

Perron¹,

Shang²,

Kraska³

2019

2019 IEEE 35th International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

show abstract

“…Methods for replacing an evaluation plan on-the-fly without significantly affecting system performance or losing intermediate results are a major focus of current research [29]. Numerous advanced techniques were proposed in the field of continuous query processing in data streams [10,37,52].…”

Section: Detection-adaptation Loopmentioning

confidence: 99%

“…Adaptive query processing (AQP) is the widely studied problem of adapting a query plan to the unstable data characteristics [29]. Multiple solutions consider traditional data-bases [5,32,15,41,33,46].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Efficient adaptive detection of complex event patterns

Kolchinsky

Schuster

2018

Proc. VLDB Endow.

View full text Add to dashboard Cite

Complex event processing (CEP) is widely employed to detect occurrences of predefined combinations (patterns) of events in massive data streams. As new events are accepted, they are matched using some type of evaluation structure, commonly optimized according to the statistical properties of the data items in the input stream. However, in many reallife scenarios the data characteristics are never known in advance or are subject to frequent on-the-fly changes. To modify the evaluation structure as a reaction to such changes, adaptation mechanisms are employed. These mechanisms typically function by monitoring a set of properties and applying a new evaluation plan when significant deviation from the initial values is observed. This strategy often leads to missing important input changes or it may incur substantial computational overhead by over-adapting.In this paper, we present an efficient and precise method for dynamically deciding whether and how the evaluation structure should be reoptimized. This method is based on a small set of constraints to be satisfied by the monitored values, defined such that a better evaluation plan is guaranteed if any of the constraints is violated. To the best of our knowledge, our proposed mechanism is the first to provably avoid false positives on reoptimization decisions. We formally prove this claim and demonstrate how our method can be applied on known algorithms for evaluation plan generation. Our extensive experimental evaluation on real-world datasets confirms the superiority of our strategy over existing methods in terms of performance and accuracy.

show abstract

“…To ensure efficient operation, a CEP engine must continuously estimate the current statistic values and, when a significant deviation is detected, adapt itself by recalculating the affected evaluation plans. Developing efficient adaptive mechanisms is considered a hard problem and a hot research topic in several fields [9,19,29,35].…”

Section: Adaptive Complex Event Processingmentioning

confidence: 99%

Join query optimization techniques for complex event processing applications

Kolchinsky

Schuster

2018

Proc. VLDB Endow.

View full text Add to dashboard Cite

Complex event processing (CEP) is a prominent technology used in many modern applications for monitoring and tracking events of interest in massive data streams. CEP engines inspect real-time information flows and attempt to detect combinations of occurrences matching predefined patterns. This is done by combining basic data items, also called "primitive events", according to a pattern detection plan, in a manner similar to the execution of multi-join queries in traditional data management systems. Despite this similarity, little work has been done on utilizing existing join optimization methods to improve the performance of CEP-based systems.In this paper, we provide the first theoretical and experimental study of the relationship between these two research areas. We formally prove that the CEP Plan Generation problem is equivalent to the Join Query Plan Generation problem for a restricted class of patterns and can be reduced to it for a considerably wider range of classes. This result implies the NP-completeness of the CEP Plan Generation problem. We further show how join query optimization techniques developed over the last decades can be adapted and utilized to provide practically efficient solutions for complex event detection. Our experiments demonstrate the superiority of these techniques over existing strategies for CEP optimization in terms of throughput, latency, and memory consumption.

show abstract

Adaptive Query Processing

Cited by 51 publications

References 119 publications

How I Learned to Stop Worrying and Love Re-optimization

How I Learned to Stop Worrying and Love Re-optimization

Efficient adaptive detection of complex event patterns

Join query optimization techniques for complex event processing applications

Contact Info

Product

Resources

About