PLAStiCC: Predictive Look-Ahead Scheduling for Continuous Dataflows on Clouds

Abstract

Abstract-Scalable stream processing and continuous dataflow systems are gaining traction with the rise of big data due to the need for processing high velocity data in near real time. Unlike batch processing systems such as MapReduce and workflows, static scheduling strategies fall short for continuous dataflows due to the variations in the input data rates and the need for sustained throughput. The elastic resource provisioning of cloud infrastructure is valuable to meet the changing resource needs of such continuous applications. However, multi-tenant cloud resources introduce yet another dimension of performance variability that impacts the application's throughput. In this paper we propose PLAStiCC, an adaptive scheduling algorithm that balances resource cost and application throughput using a prediction-based lookahead approach. It not only addresses variations in the input data rates but also the underlying cloud infrastructure. In addition, we also propose several simpler static scheduling heuristics that operate in the absence of accurate performance prediction model. These static and adaptive heuristics are evaluated through extensive simulations using performance traces obtained from public and private IaaS clouds. Our results show an improvement of upto 20% in the overall profit as compared to the reactive adaptation algorithm.

Fog over Virtualized IoT: New Opportunity for Context-Aware Networked Applications and a Case Study

“…In this section, we aim to focus on the resource management of Cloud/Fog-based distributed computing architectures for the energy-efficient support of real-time Big Data Streaming (BDS) applications by resource-limited wireless devices [13][14][15][16]. In detail, the S4 and DSstream management frameworks in [13,14] perform dynamic resource scaling of the available virtualized resources by explicitly accounting for the delay-sensitive nature of the streaming workload offloaded by proximate wireless devices.…”

“…In detail, the S4 and DSstream management frameworks in [13,14] perform dynamic resource scaling of the available virtualized resources by explicitly accounting for the delay-sensitive nature of the streaming workload offloaded by proximate wireless devices. The time stream and PLAstiCC resource orchestrator in [15,16] integrate dynamic server consolidation and inter-server live VM migration. Overall, as in our contribution, the shared goals of [13][14][15][16] are (i) the provision of real-time computing support to BDS applications run by resource-limited wireless devices through the exploitation of the virtualized resources made available by proximate FNs; (ii) minimization of the overall inter/intra-data center computing-plus-networking energy consumption under BDS applications.…”

Fog over Virtualized IoT: New Opportunity for Context-Aware Networked Applications and a Case Study

“…In this section we aim to focusing on the resource management of Cloud/Fog-based distributed computing architectures for the energy-efficient support of real-time Big Data Streaming (BDS) applications by resource-limited wireless devices [13][14][15][16]. In detail, the S4 and DS-streams management frameworks in [13] and [14] perform dynamic resource scaling of the available virtualized resources by explicitly accounting for the delay-sensitive nature of the streaming workload offloaded by proximate wireless devices.…”

“…In detail, the S4 and DS-streams management frameworks in [13] and [14] perform dynamic resource scaling of the available virtualized resources by explicitly accounting for the delay-sensitive nature of the streaming workload offloaded by proximate wireless devices. The Time Stream and PLAstiCC resource orchestrator in [15] and [16] integrate dynamic server consolidation and inter-server live VM migration. Overall, like our contribution, the shared goals of [13][14][15] and [16] are: (i) the provisioning of real-time computing support to BDS applications run by resource-limited wireless devices through the exploitation of the virtualized resources done available by proximate FNs; and, (ii) the minimization of the overall inter/intra-data center computing-plus-networking energy consumption under BDS applications.…”

“…The Time Stream and PLAstiCC resource orchestrator in [15] and [16] integrate dynamic server consolidation and inter-server live VM migration. Overall, like our contribution, the shared goals of [13][14][15] and [16] are: (i) the provisioning of real-time computing support to BDS applications run by resource-limited wireless devices through the exploitation of the virtualized resources done available by proximate FNs; and, (ii) the minimization of the overall inter/intra-data center computing-plus-networking energy consumption under BDS applications. However, unlike our contribution, we point out that: (i) the resource management techniques developed in [13] and [14] are not capable to self-tune to the (possibly, un-predictable) time fluctuation of the workload to be processed; and, (ii) the management approaches pursued in [15] and [16] do not guarantee hard limits on the resulting per-task execution times.…”