Efficient adaptive detection of complex event patterns

Abstract

Complex event processing (CEP) is widely employed to detect occurrences of predefined combinations (patterns) of events in massive data streams. As new events are accepted, they are matched using some type of evaluation structure, commonly optimized according to the statistical properties of the data items in the input stream. However, in many reallife scenarios the data characteristics are never known in advance or are subject to frequent on-the-fly changes. To modify the evaluation structure as a reaction to such changes, adaptation mechanisms are employed. These mechanisms typically function by monitoring a set of properties and applying a new evaluation plan when significant deviation from the initial values is observed. This strategy often leads to missing important input changes or it may incur substantial computational overhead by over-adapting.In this paper, we present an efficient and precise method for dynamically deciding whether and how the evaluation structure should be reoptimized. This method is based on a small set of constraints to be satisfied by the monitored values, defined such that a better evaluation plan is guaranteed if any of the constraints is violated. To the best of our knowledge, our proposed mechanism is the first to provably avoid false positives on reoptimization decisions. We formally prove this claim and demonstrate how our method can be applied on known algorithms for evaluation plan generation. Our extensive experimental evaluation on real-world datasets confirms the superiority of our strategy over existing methods in terms of performance and accuracy.

Event Trend Aggregation Under Rich Event Matching Semantics

“…ZStream translates an event query into an operator tree optimized using rewrite rules. Similarly, Kolchinsky et al apply traditional join-query optimization techniques to evaluate event sequence and Kleene closure queries [22] and adapt an optimization plan to changing data characteristics [21]. E-Cube employs hierarchical event stacks to share events across different event queries.…”

“…Others implement hardware-based CEP on FPGAs at gigabit wire speed [48]. The expressive power of these approaches is limited since they do not support Kleene closure [29], nor aggregation [13,21,22,36,38,48], nor various event matching semantics [13-15, 21, 29, 33, 36, 38, 48]. In contrast, SASE and Flink support all these language constructs but they do not design any optimization techniques for event trend aggregation.…”

Join query optimization techniques for complex event processing applications

“…Multiple works have presented methods for constructing NFAs with out-of-order processing support. W.l.o.g., we will use the Lazy NFA mechanism, a chain-structured NFA introduced in [28,29] and capable of following a specified evaluation order.…”

“…This section describes how a pattern of each of the aforementioned types can be represented and detected as either a pure conjunctive pattern or their union. To that end, we will utilize some of the ideas from [28]. Note that the transformations presented below are only applied for the purpose of plan generation, that is, no actual conversion takes place during execution on a data stream.…”

“…Due to an extremely high arrival rate of B , its processing will likely be postponed to the latest step in the plan, which is also the desired strategy for the original pattern in this case. B will then be replaced with B in the resulting plan, and the missing Kleene closure operator will be added in the respective stage (by modifying an edge type for a NFA [28] or a node type for a tree [35]), thus producing a valid plan for detecting the original pattern. We will now formally prove the correctness of the above construction.…”