Supporting Strong Reliability for Distributed Complex Event Processing Systems

Abstract

Many application classes such as monitoring applications, involve processing a massive amount of data from a possibly huge number of data sources. Complex Event Processing (CEP) has evolved as the paradigm of choice to determine meaningful situations (complex events) by performing stepwise correlation over event streams. To keep up with the high scalability demands of growing input streams, recent approaches distribute event correlation over several correlation nodes. However, already a failure of a single correlation node impacts the correctness of the final correlation result. In this paper, we illustrate the importance of a strong reliability semantics for CEP in the context of a monitoring application in a distributed production environment. Strong reliability ensures each complex event is detected and delivered exactly once to each application entity, and cannot be guaranteed by the naive application of established replication principles. We present a replication scheme which ensures strong reliability in an asynchronous system model and can be applied to an arbitrary distributed CEP system. The algorithm tolerates f simultaneous failures by introducing f additional replicas for each correlation node. We prove correctness as well as evaluate the overhead introduced by the algorithm. Results show, that the overhead scales linearly with the number of deployed replicas and the node failure rate.

Moving range queries in distributed complex event processing

“…However, the distributed processing of multiple dependent operators is not addressed by existing work. This way, it becomes possible to apply many optimizations used for distributed CEP systems [20,22,24,29] also in the context of distributed range queries.…”

Rollback-recovery without checkpoints in distributed event processing systems

“…As the simulated operator topology, we chose n-ary trees with a depth of 3 for n = 1 to 5, with the root operator connected to 1 consumer and each leave operator connected to 1 source. Event sources produce events with a frequency of 1 event / ms. We compare the overhead with the messaging overhead that would have been caused by duplicate events in the CEP-optimized active replication approach [20] developed by Völz et al We assume a low replication factor of 2 and the best case scenario for the leader election (only one leader at a time), leading to an overhead that approximately equates to the number of events sent through the network regularly, and neglect the overhead that the leader election would cause. Figure 10 shows how much additional data is sent over the network within 5 minutes.…”

An Efficient Data Dissemination Approach for Cloud Monitoring

“…Monitoring Data Processing: Monitoring applications often involves processing a massive amount of data from a possibly huge number of data sources [19]. CEP [6] has evolved as the paradigm of choice to determine meaningful situations (complex events) for decision making by performing stepwise correlation over event streams in many domains, such as processing of environmental sensor data, trades in financial markets and RSS web feeds [19,20]. In [21], a complex event language that significantly extends existing event languages to meet the needs of a range of RFID enabled monitoring applications is introduced first, then a query plan-based approach and some optimization techniques are used to efficiently implementing this language.…”