High-performance nested CEP query processing over event streams

Abstract

Ahstract-Complex event processing (CEP) over event streams has become increasingly important for real-time applications ranging from health care, supply chain management to busi ness intelligence. These monitoring applications submit complex queries to track sequences of events that match a given pattern.As these systems mature the need for increasingly complex nested sequence query support arises, while the state-of-art CEP systems mostly support the execution of flat sequence queries only. To assure real-time responsiveness and scalability for pattern detection even on huge volume high-speed streams, efficient processing techniques must be designed. In this paper, we first analyze the prevailing nested pattern query processing strategy and identify several serious shortcomings. Not only are substantial subsequences first constructed just to be subse quently discarded, but also opportunities for shared execution of nested subexpressions are overlooked. As foundation, we introduce NEEL, a CEP query language for expressing nested CEP pattern queries composed of sequence, negation, AND and OR operators. To overcome deficiencies, we design rewriting rules for pushing negation into inner subexpressions. Next, we devise a normalization procedure that employs these rules for flattening a nested complex event expression. To conserve CPU and memory consumption, we propose several strategies for efficient shared processing of groups of normalized NEEL subexpressions. These strategies include prefix caching, suffix clustering and customized "bit-marking" execution strategies. We design an optimizer to partition the set of all CEP sub expressions in a NEEL normal form into groups, each of which can then be mapped to one of our shared execution operators. Lastly, we evaluate our technologies by conducting a performance study to assess the CPU processing time using real-world stock trades data. Our results confirm that our NEEL execution in many cases performs 100 fold faster than the traditional iterative nested execution strategy for real stock market query workloads.

New parallel processing strategies in complex event processing systems with data streams

“…[10][11][12]15 This approach is both orthogonal and complementary to the previous approaches. Our approach can be used even to parallelize a pattern operator with CEP queries that contain no ''PARTITION BY'' clauses.…”

“…An AND operator 15 takes the event types as input, and events occur within a specified time window without a specified time order…”

“…[8][9][10][11][12][13][14][15] Most of the conventional CEP systems are centralized, running on a single machine. With the need for high performance, distributed CEP systems are developed by distributing the detection logic over a network of operators, where individual operators can be a bottleneck, requiring operator parallelization.…”

“…Such complex pattern queries involve nests of sequences (SEQ) and conjunctions (AND), which can have negative event type(s), and combinations of them. 11,13,15,39 The volume and input rates of the data, however, would become large, similar to event stream processing, especially for big data applications. 40,41 Increasing the size of the time window of an operator or the input rate of a stream may cause bottlenecks, which gives rise to query results of poor quality, losing the quality-of-service (QoS) guarantees of the system.…”

“…[2][3][4][5][6][7] Another technology is about complex event processing (CEP). [8][9][10][11][12][13][14][15] CEP is a method of processing and analyzing the data streams of information by making use of patterns over sequential primitive events for detecting and reporting composite events. 16 Because of advantages such as an expressive rule language and an efficient event detection model, CEP systems have been applied extensively in both academic circles and industry in recent times.…”