Khaos: Dynamically Optimizing Checkpointing for Dependable Distributed Stream Processing

Abstract

Distributed Stream Processing systems are becoming an increasingly essential part of Big Data processing platforms as users grow ever more reliant on their ability to provide fast access to new results. As such, making timely decisions based on these results is dependent on a system's ability to tolerate failure. Typically, these systems achieve fault tolerance and the ability to recover automatically from partial failures by implementing checkpoint and rollback recovery. However, owing to the statistical probability of partial failures occurring in these distributed environments and the variability of workloads upon which jobs are expected to operate, static configurations will often not meet Quality of Service constraints with low overhead.In this paper we present Khaos, a new approach which utilizes the parallel processing capabilities of cloud orchestration technologies for the automatic runtime optimization of fault tolerance configurations in Distributed Stream Processing jobs. Our approach employs three subsequent phases which borrows from the principles of Chaos Engineering: establish the steadystate processing conditions, conduct experiments to better understand how the system performs under failure, and use this knowledge to continuously minimize Quality of Service violations. We implemented Khaos prototypically together with Apache Flink and demonstrate its usefulness experimentally.

Daedalus: Self-Adaptive Horizontal Autoscaling for Resource Efficiency of Distributed Stream Processing Systems

“…The Traffic Monitoring job is an IoT use case that calculates the average speed of moving vehicles in a particular radius adapted from the IoT Vehicles Experiment [9]. The job reads JSON vehicle events from a Kafka source, filters out events not contained within a radius of interest, calculates the average speed of vehicles in a ten second tumbling window, and enriches the vehicle information before outputting to a Kafka sink.…”

“…The Yahoo Streaming Benchmark workload is taken from realistic online advertising click-through rate data 3 . Lastly, the traffic monitoring workload was generated based on the TAPASCologne scenario and SUMO to simulate realistic traffic patterns in the city of Berlin [9]. Each job was benchmarked to determine the maximum throughput achievable with 12 workers.…”

Probabilistic Time Series Forecasting for Adaptive Monitoring in Edge Computing Environments

“…The metrics of the host machine are eventually gathered via Prometheus queries and saved to file. As a source for publishing to the message queue, we employ the IoT Vehicles experiment dataset created and published in [28], which reports amounts of moving cars at intervals of 1 second. At each point in time, the corresponding vehicle amount is read and used for publishing the same amount of messages to the message queue.…”