QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

Abstract

Abstract-Many emerging on-line data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real-time. Distributed stream processing systems allow in-network stream processing to achieve better scalability and quality-of-service (QoS) provision. In this paper we present Synergy, a novel distributed stream processing middleware that provides automatic sharingaware component composition capability. Synergy enables efficient reuse of both result streams and processing components, while composing distributed stream processing applications with QoS demands. It provides a set of fully distributed algorithms to discover and evaluate the reusability of available result streams and processing components when instantiating new stream applications. Specifically, Synergy performs QoS impact projection to examine whether the shared processing can cause QoS violations on currently running applications. The QoS impact projection algorithm can handle different types of streams including both regular traffic and bursty traffic. If no existing processing components can be reused, Synergy dynamically deploys new components at strategic locations to satisfy new application requests. We have implemented a prototype of the Synergy middleware and evaluated its performance on both PlanetLab and simulation testbeds. The experimental results show that Synergy can achieve much better resource utilization and QoS provisioning than previously proposed schemes, by judiciously sharing streams and components during application composition.

A Decentralized Approach to Network-Aware Service Composition

“…3 adopts a heuristic procedure that tries to determine a "good enough" composition, without a strict guarantee that an optimal solution will be found. The procedure is based on a greedy approach similar to the ones proposed in [9,19], which resolves services in Σ W on a stepby-step basis, starting from the initial node of W : indeed, a required service σ is bound to a matching concrete service only if all preceding services in Pred (σ) have already been resolved (line 7). The rationale for this mechanism is that only when Pred (σ) is fully resolved, it is possible to know the worst case increment to the path length caused by resources in the set of known candidates for σ, and to select the one causing the minimal increment (lines 7-9).…”

“…The proposed approach supports parallel service flows, but presents some limits with respect to composite complexity, network density, and service demand. Repantis et al [19] focus on stream processing applications and propose a service composition approach where the component discovery phase exploits a structured P2P network and precedes the probe-based composition phase. We also mention the work in [15], that proposes DANS, a decentralized multimedia workflow processing system.…”

“…Rather, they motivate the need of decentralized and self-adaptive approaches to achieve an adequate degree of robustness, resiliency and scalability. Approaches of this kind have already been proposed [9,19], mostly based on the assumption of a decentralized orchestration of the application workflow, using structured or unstructured P2P network architectures.…”

Scaling a monitoring infrastructure for the Akamai network

“…Borealis [1], as a continuation of the Aurora Project focused on the challenges related to implementing such a system in a distributed fashion, with particular emphasis in load shedding and fault-tolerance. Synergy [10] has focused on composing distributed stream processing applications, while paying attention to their endto-end Quality of Service requirements.…”

A trusted routing algorithm based on D-S evidence theory combined with physical layer security

“…With the development of wireless communication technology, a lot of research introduce the physical layer security technologies to improve the routing effect, and these have become a hot topic of research in recent years [1][2] [3]. In this paper, we try to propose a trusted routing algorithm to improve the success rate of routing and the performance of the QoS metrics [4].…”