Replica placement for high availability in distributed stream processing systems

Repantis, Thomas; Kalogeraki, Vana

doi:10.1145/1385989.1386012

Cited by 11 publications

(10 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proactive migration [34] can guard against QoS violations. On the other hand, predictive failure management [35], and availability-aware placement [36] can protect against component or node failures. To offer fault tolerance, either reactive or proactive failure recovery schemes can be used [10].…”

Section: Discussionmentioning

confidence: 99%

QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

Repantis

Kalogeraki

2009

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-Many emerging on-line data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real-time. Distributed stream processing systems allow in-network stream processing to achieve better scalability and quality-of-service (QoS) provision. In this paper we present Synergy, a novel distributed stream processing middleware that provides automatic sharingaware component composition capability. Synergy enables efficient reuse of both result streams and processing components, while composing distributed stream processing applications with QoS demands. It provides a set of fully distributed algorithms to discover and evaluate the reusability of available result streams and processing components when instantiating new stream applications. Specifically, Synergy performs QoS impact projection to examine whether the shared processing can cause QoS violations on currently running applications. The QoS impact projection algorithm can handle different types of streams including both regular traffic and bursty traffic. If no existing processing components can be reused, Synergy dynamically deploys new components at strategic locations to satisfy new application requests. We have implemented a prototype of the Synergy middleware and evaluated its performance on both PlanetLab and simulation testbeds. The experimental results show that Synergy can achieve much better resource utilization and QoS provisioning than previously proposed schemes, by judiciously sharing streams and components during application composition.

show abstract

Section: Discussionmentioning

confidence: 99%

QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

Repantis

Kalogeraki

2009

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…Different authors [5,13,23] studied the problem of increasing the availability of a data stream processing system by an optimized placement of replicas to hosts. Repantis et al [23] presented a scheme for replica placement considering also performance constraints like end to end latency by intelligently selecting hosts for primary and replica placement.…”

Section: Replica Placementmentioning

confidence: 99%

“…Repantis et al [23] presented a scheme for replica placement considering also performance constraints like end to end latency by intelligently selecting hosts for primary and replica placement. The system ZEN [5] models for their placement approach three different zones of availability (according to the bathtub curve for host availability) inside the cluster and tries to assign the most important operators to the hosts with the highest availability.…”

Section: Replica Placementmentioning

confidence: 99%

An adaptive replication scheme for elastic data stream processing systems

Heinze¹,

Zia

Krahn

et al. 2015

Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

View full text Add to dashboard Cite

A major challenge for cloud-based systems is to be fault tolerant to cope with an increasing probability of faults in cloud environments. This is especially true for in-memory computing solutions like data stream processing systems, where a single host failure might result in an unrecoverable information loss.In state of the art data streaming systems either active replication or upstream backup are applied to ensure fault tolerance, which have a high resource overhead or a high recovery time respectively. This paper combines these two fault tolerance mechanisms in one system to minimize the number of violations of a user-defined recovery time threshold and to reduce the overall resource consumption compared to active replication. The system switches for individual operators between both replication techniques dynamically based on the current workload characteristics. Our approach is implemented as an extension of an elastic data stream processing engine, which is able to reduce the number of used hosts due to the smaller replication overhead. Based on a real-world evaluation we show that our system is able to reduce the resource usage by up to 19% compared to an active replication scheme.

show abstract

“…Traditionally, high availability mechanism (HA for short) which includes replica backup, fail recovery and other policies is dedicated and tightly coupled via specific protocols in data stream processing system, such as TelegraphCQ [3,4], Borealis [5,6], Synergy [7,8] and CLASP [9][10][11]. However, there are some limitations.…”

Section: Introductionmentioning

confidence: 99%

Stream-oriented Availability Services for Endpoint-to-endpoint Data Transmission

Ding

Han

Zhao

et al. 2012

2012 International Conference on Cloud and Service Computing

View full text Add to dashboard Cite

Continuous real-time data stream show its significance in many applications under Cloud computing. Traditional high availability mechanism for data stream processing systems is dedicated and tightly coupled, and brings difficulties on maintenance or extension if requirements change. In this paper, basic endpoint-to-endpoint transmission is separated as general services from particular systems to independently provide no-loss guarantee of upstream data, through which high level extended HA mechanism can be realized conveniently. Comprehensive experiments illustrate that the overhead of the services is quite small and the tradeoff between guarantee and overheads is acceptable.

show abstract

Replica placement for high availability in distributed stream processing systems

Cited by 11 publications

References 43 publications

QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

An adaptive replication scheme for elastic data stream processing systems

Stream-oriented Availability Services for Endpoint-to-endpoint Data Transmission

Contact Info

Product

Resources

About