Dynamic datacenter resource provisioning for high-performance distributed stream processing with adaptive fault-tolerance

Abstract

A growing number of applications require continuous processing of high-throughput data streams, e.g., financial analysis, network traffic monitoring, or Big Data analytics in smart cities. Stream processing applications typically have explicit quality-of-service requirements; yet, due to the high time-variability of stream characteristics, it is inefficient and sometimes impossible to statically allocate all the resources needed to guarantee application SLAs. In this work, we present DARM, a novel middleware for adaptive replication that trades fault-tolerance for increased capacity during load spikes and provides guaranteed upper-bounds on information loss in case of failures.