Building user-defined runtime adaptation routines for stream processing applications

Abstract

Stream processing applications are deployed as continuous queries that run from the time of their submission until their cancellation. This deployment mode limits developers who need their applications to perform runtime adaptation, such as algorithmic adjustments, incremental job deployment, and application-specific failure recovery. Currently, developers do runtime adaptation by using external scripts and/or by inserting operators into the stream processing graph that are unrelated to the data processing logic. In this paper, we describe a component called orchestrator that allows users to write routines for automatically adapting the application to runtime conditions. Developers build an orchestrator by registering and handling events as well as specifying actuations. Events can be generated due to changes in the system state (e.g., application component failures), built-in system metrics (e.g., throughput of a connection), or custom application metrics (e.g., quality score). Once the orchestrator receives an event, users can take adaptation actions by using the orchestrator actuation APIs. We demonstrate the use of the orchestrator in IBM's System S in the context of three different applications, illustrating application adaptation to changes on the incoming data distribution, to application failures, and on-demand dynamic composition.

Towards a flexible data stream analytics platform based on the GCM autonomous software component technology

“…Floe, like our solution based on GCM, adopts a component model of application execution so that a single centralized dataflow orchestrator is not a bottleneck. [12] based upon the System S IBM's stream analytics platform features the same goals as us but offers a different, not integrated, approach from what we intend to do: developers program in advance (in ORCA) an external (and centralized) orchestrator that corresponds to their applications management policy. They express which events (pertaining to either functional or non-functional aspects) are of interest and how the application should adapt upon the occurrence of these events.…”

“…Functional layer GCM (Grid Component Model) [1] is a hierarchical component-oriented model for building parallel and distributed software. It is a framework that uses autonomous components [5], [6] [11], [5] [6], [7], [8], [9] [3], [12], [10] as the building block for building applications, each of which communicates and sends messages (in the form of remote method or service calls) to each other to accomplish the execution of the defined tasks. By autonomous, we mean that the GCM programming model enforces to define loosely coupled components that do not share memory nor control, and thus can be considered as independent entities.…”

Runtime‐aware adaptive scheduling in stream processing

“…For the fast nodes, we keep their initial tasks the same and assign more tasks to them if they have extra ability to process tuples. The results are illustrated by the following rules (lines [16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32]. For the node i in task-mapping-table, (1) If the task j is initially deployed on node i and it has not been reassigned to another node, the node i will have the task j (line [17][18][19][20][21][22].…”

A middleware for managing dynamic software adaptation

“…supports both parameter and compositional adaptation. One of the most widespread approaches in software engineering is to separate the application's business logic from adaptation logic [8] [4] [9] [6]. Following this trend, we provide an API where adaptation engineers, the ones responsible for building the application´s adaptation plans, can register for notifications of system events (e.g., availability of new devices, component failure reports, and other application-specific events) so as to create specific adaptation plans and tasks, as means of reacting to those events.…”