Stream Data Load Prediction for Resource Scaling Using Online Support Vector Regression

Abstract

A distributed data stream processing system handles real-time, changeable and sudden streaming data load. Its elastic resource allocation has become a fundamental and challenging problem with a fixed strategy that will result in waste of resources or a reduction in QoS (quality of service). Spark Streaming as an emerging system has been developed to process real time stream data analytics by using micro-batch approach. In this paper, first, we propose an improved SVR (support vector regression) based stream data load prediction scheme. Then, we design a spark-based maximum sustainable throughput of time window (MSTW) performance model to find the optimized number of virtual machines. Finally, we present a resource scaling algorithm TWRES (time window resource elasticity scaling algorithm) with MSTW constraint and streaming data load prediction. The evaluation results show that TWRES could improve resource utilization and mitigate SLA (service level agreement) violation.

Phoebe: QoS-Aware Distributed Stream Processing through Anticipating Dynamic Workloads

“…3) TWRES: We employ a second dynamic scaling baseline inspired from recent related work. Precisely, we use the resource scaling algorithm (TWRES) proposed in [10] for Spark streaming jobs. Similar to Phoebe, this algorithm requires profiling data, and scales a data processing application under consideration of workload forecasts, a performance model for the maximum processing capacity of individual scaleouts, and formulated latency constraints.…”

“…1) Top Speed Windowing (TSW) Experiment: For the first experiment, a DSP job was selected from the official Flink repository 10 . It was modified so that sources consumed events from and sinks published results to separate Apache Kafka topics.…”

Evaluation of Load Prediction Techniques for Distributed Stream Processing

“…6) Avazu: This dataset is created by using a click-through rate prediction dataset from Kaggle 9 , aggregating the clicks per hour over time, and linearly interpolating between the aggregated values to obtain different sampling rates.…”

“…However, none has been directly compared under our defined requirements for performing TSF in DSP systems, such as minimal configuration and limited model inputs. In the context of DSP, TSF methods have been used in diverse forms and for varying reasons [9], [35]- [38]. While previous works successfully apply a selected method to a concrete problem, to the best of our knowledge, there is no related work that compares multiple TSF methods for DSP.…”

“…Prominent applications within this domain include: the automated dynamic scaling of resources which aims to reduce over-and under-provisioning, minimizing operating costs and preventing possible reductions in the service quality [6]- [9]; The live migration of functionality/state across the network where cluster metrics are collected, processed, and scanned for anomalies which might reveal performance degradations and/or signs of component failure [10]- [12]; and the automatic system tuning where dynamic runtime adjustment of system configurations are performed in order to improve overall system availability and reliability [13]- [15]. The majority of these approaches rely on coarse-grained metrics to reactively make remediation decisions.…”