Stream-oriented Availability Services for Endpoint-to-endpoint Data Transmission

Abstract

Continuous real-time data stream show its significance in many applications under Cloud computing. Traditional high availability mechanism for data stream processing systems is dedicated and tightly coupled, and brings difficulties on maintenance or extension if requirements change. In this paper, basic endpoint-to-endpoint transmission is separated as general services from particular systems to independently provide no-loss guarantee of upstream data, through which high level extended HA mechanism can be realized conveniently. Comprehensive experiments illustrate that the overhead of the services is quite small and the tradeoff between guarantee and overheads is acceptable.

Feature‐based high‐availability mechanism for quantile tasks in real‐time data stream processing

“…In each configuration, for task QT , feature‐based HA and replica‐based HA are realized as Figure (a and b), respectively, with the help of HAC‐node module. As an endpoint‐to‐endpoint stream connector proposed in our previous work, HAC‐node provides basic services (e.g. status detection and configuration management) and programming primitives, through which customized HA functions can be implemented.…”

“…Here, we classify the HA for tasks from two perspectives [11,12,[17][18][19][20][21][22][23]. Task is also termed as processing element [11][12][13] or node [14] in other related works.…”

“…Two high-availability mechanisms for a quantile task. 865 FEATURE-BASED HIGH-AVAILABILITY MECHANISM FOR QUANTILE TASKS endpoint-to-endpoint stream connector proposed in our previous work, HAC-node [14] provides basic services (e.g. status detection and configuration management) and programming primitives, through which customized HA functions can be implemented.…”

“…Physical environment is Dell PowerEdge TM rack servers, which include AMD 16 core CPU, 32GB DDR2 RAM, 500GB SATA disk with operating system CentOS 5.5 × 86 64bit. All the experiments are completed on our VINCA C oncurrent C onnection S tream (VINCA‐CCS) system. That is a distributed data stream processing system developed by our team using C/C++ on Linux environment.…”

“…There are more than 60 distributed tasks in the system, which can be divided into 11 distinct business jobs orthogonally. Here, a task is the logical processing unit of the business job, which is also termed as processing element or node in other works; a business job is online domain computation of grid transportation, in which continuous tuples are received as data stream concurrently and processed by multiple tasks collaboratively for specific purpose. These business jobs currently are fake‐plate detection, copy‐plate detection, black‐list detection, accompanied‐car analysis, traffic‐flow analysis and travel‐time analysis.…”