A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Abstract

Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced e.g. in the domain of the Internet of ings. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enables SP systems to process these streams with continuously high quality of service. e current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes speci c assumptions and focuses on particular aspects of the problem. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. erefore, in this survey, we study the literature and develop a classi cation of current methods for both parallelization and elasticity in SP systems. or even di erent processing nodes in a shared-nothing cloud-based infrastructure. Frequent state synchronization must not hamper parallel processing, while the processing results have to remain consistent. Research proposes di erent approaches for parallel, stateless and stateful SP. ey di er in assumptions about the operator functions and state externalization mechanisms an SP system supports.is led to the development of a broad range of parallelization approaches tackling di erent problem cases.Second, how to continuously adapt the level of parallelization when the conditions of the SP operators, e.g. the workload or resources available, change at runtime. On the one hand, an SP system always needs enough resources to process the input data streams with a satisfying quality of service (QoS), e.g. latency or throughput. On the other hand, continuous provisioning of computing resources for peak workloads wastes resources at o -peak hours. us, an elastic SP system scales its resources according to the current need. Cloud computing provides on-demand resources to realize such elasticity [9]. e pay-as-you-go business model of cloud computing allows to cut costs by dynamically adapting the resource reservations to the needs of the SP system. It is challenging to strive the right balance between resource over-provisioning-which is costly, but is robust to workload uctuations-and on-demand scaling-which is cheap, but is vulnerable to sudden workload peaks. To this end, academia and industry developed elasticity methods. Again, they di er in their optimization objectives and assumptions about the operator parallelization model employed, the target system architecture, state management as well as timing and methodology.While there are many works that propose methods and solutions for speci c parallelization and elasticity problems in SP systems, there is a severe lack of overview, comparison, and classi cation of these methods. When we investigate...

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

“…Dayarathna et al [34] focus on the system architecture and use cases of stream processing platforms and only briefly discuss performance scalability. A more recent survey [99] proposes a classification for parallelization and elasticity methods in stream processing systems but lacks any discussion on automatic parameter tuning. Autonomic computing brings together the work of computer science and control communities with the purpose of developing autonomic systems, i.e., systems that can manage themselves automatically.…”

Orchestrating the Development Lifecycle of Machine Learning-based IoT Applications

“…This aggregation requires application of various data summarization techniques [71] including sampling, sketching, histograms, wavelets and adaptation of these techniques to meet the constraints of the hardware and the time-varying channel conditions. Henriette et al [280] investigated the state-of-theart stream processing systems that can be used to implement these data summarization techniques and execute them in a parallel and elastic manner. However, it still requires a lot of effort to develop new data summarization techniques and stream processing systems to handle the difficulty of processing high volumes data from various sources with multi-modality.…”

Scalable and Reliable Multi-dimensional Sensor Data Aggregation in Data Streaming Architectures

“…As most approaches applying stream processing, our presented approach is primarily based on data parallelism [2], meaning that (sub)topologies exist in multiple instances, each processing a portion of the data. Pipe-and-Filter frameworks such as TeeTime [25] employ task parallelism, where the individual filters (operators) are executed in parallel.…”

“…Stream processing [1,2] has evolved as a paradigm to process and analyze continuous streams of data, for example, coming from IoT sensors. The rapid development of stream processing engines [3] over the last years has paved the way for applications that process data exclusively online, i.e., as soon as it is recorded.…”