Online and transparent self-adaptation of stream parallel patterns

Abstract

Several real-world parallel applications are becoming more dynamic and long-running, demanding online (at run-time) adaptations. Stream processing is a representative scenario that computes data items arriving in real-time and where parallel executions are necessary. However, it is challenging for humans to monitor and manually self-optimize complex and long-running parallel executions continuously. Moreover, although high-level and structured parallel programming aims to facilitate parallelism, several issues still need to be addressed for improving the existing abstractions. In this paper, we extend self-adaptiveness for supporting autonomous and online changes of the parallel pattern compositions. Online self-adaptation is achieved with an online profiler that characterizes the applications, which is combined with a new self-adaptive strategy and a model for smooth transitions on reconfigurations. The solution provides a new abstraction layer that enables application programmers to define non-functional requirements instead of hand-tuning complex configurations. Hence, we contribute with additional abstractions and flexible self-adaptation for responsiveness at run-time. The proposed solution is evaluated with applications having different processing characteristics, workloads, and configurations. The results show that it is possible to provide additional abstractions, flexibility, and responsiveness while achieving performance comparable to the best static configuration executions.

Self‐adaptation on parallel stream processing: A systematic review

“…As recommended by Kitchenham's guidelines, 35 the research questions of this literature review were separated into facets related to three main aspects: stream processing, self‐adaptiveness, and parallelism. We defined five control studies 36‐40 that were previously known as relevant primary studies from our previous works 8,15,16,41,42 . Pilot searches were conducted on Scopus with the different terms and synonyms for testing if the string found the control studies.…”

“…We defined five control studies [36][37][38][39][40] that were previously known as relevant primary studies from our previous works. 8,15,16,41,42 Pilot searches were conducted on Scopus with the different terms and synonyms for testing if the string found the control studies. Thus, we converged to a search string evinced in Table 2 with the terms of the three main aspects separated by boolean ANDs, where these terms searched studies' titles, abstracts, and keywords.…”

“…Figure 1 shows a farm with one fissioned (a.k.a replicated) stage, which represents a graph composition or application topology 41 . Such a graph composition is suitable for embarrassingly parallel computation that easily executes in parallel.…”

“…Figure 1 shows a farm with one fissioned (a.k.a replicated) stage, which represents a graph composition or application topology. 41 Such a graph composition is suitable for embarrassingly parallel computation that easily executes in parallel. However, there is a trend in software applications to be more complex with several functions that can be decomposed using parallel patterns, resulting in robust graph compositions.…”