Adaptive Control of Extreme-scale Stream Processing Systems

Amini, Lisa; Jain, Navendu; Sehgal, A.; Silber, J.; Verscheure, Olivier

doi:10.1109/icdcs.2006.13

Cited by 109 publications

(92 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several solutions have been presented for operator placement, each one targeting a different scenario (e.g., large scale distributed systems vs. local area network or clusters) and focusing on different goals (e.g., load balancing vs. minimization of delay or bandwidth usage) [164][165][166][167]35,28,[168][169][170][171][172][173][174]. In the context of stream reasoning, these works present one significant limitation that needs to be addressed in future research: they only focus on streaming data and do not consider the presence of stored data or background knowledge.…”

Section: Distributed Datamentioning

confidence: 99%

Streaming the Web: Reasoning over dynamic data

Margara

Urbani

Harmelen

et al. 2014

Journal of Web Semantics

111

View full text Add to dashboard Cite

a b s t r a c tIn the last few years a new research area, called stream reasoning, emerged to bridge the gap between reasoning and stream processing. While current reasoning approaches are designed to work on mainly static data, the Web is, on the other hand, extremely dynamic: information is frequently changed and updated, and new data is continuously generated from a huge number of sources, often at high rate. In other words, fresh information is constantly made available in the form of streams of new data and updates.Despite some promising investigations in the area, stream reasoning is still in its infancy, both from the perspective of models and theories development, and from the perspective of systems and tools design and implementation.The aim of this paper is threefold: (i) we identify the requirements coming from different application scenarios, and we isolate the problems they pose; (ii) we survey existing approaches and proposals in the area of stream reasoning, highlighting their strengths and limitations; (iii) we draw a research agenda to guide the future research and development of stream reasoning. In doing so, we also analyze related research fields to extract algorithms, models, techniques, and solutions that could be useful in the area of stream reasoning.

show abstract

Section: Distributed Datamentioning

confidence: 99%

Streaming the Web: Reasoning over dynamic data

Margara

Urbani

Harmelen

et al. 2014

Journal of Web Semantics

111

View full text Add to dashboard Cite

show abstract

“…With the exception of [17], these placement schemes require a centralized controller to recompute the placement of the entire operator graph in order to respond to dynamic changes in the environment such as the introduction of new operators, changes in the network data rates, and changes in the availability of machines and network links. In [17], rather than recomputing an optimal placement in response to bursty data rates, a centralized controller jointly optimizes the input and output rates of operators, as well as their instantaneous processing rates. The global optimization scheme does need to be re-run however, when new operators need to be deployed, and when existing operators expire.…”

Section: Query Operator Placement In Distributed Streammentioning

confidence: 99%

“…This approach provides theoretical analysis of a centralized placement algorithm that minimizes the total cost of computation as well as communication, but does not consider how the algorithm will respond to dynamic changes during runtime. A global optimization scheme for maximizing the weighted throughput of all queries in the system is proposed in [17]. Weights are provided as input and represent the importance or priority of a query operator.…”

Section: Query Operator Placement In Distributed Streammentioning

confidence: 99%

Biologically-Inspired Distributed Middleware Management for Stream Processing Systems

Lakshmanan¹,

Strom²

2008

Middleware 2008

View full text Add to dashboard Cite

Abstract. We present a decentralized and dynamic biologically-inspired algorithm for placing dataflow graphs composed of stream processing tasks onto a distributed network of machines, while minimizing the endto-end latency. Our algorithm responds on-the-fly to placement requests of new flow graphs or to modifications of an already running stream processing flow graph, and dynamically adapts to changes in performance characteristics such as message rates or service times as well as to changes in processor availability or link performance during runtime. Our algorithm is derived by analogy to pheromone-based cooperation between ants to fulfill goals such as food discovery. We have conducted extensive simulation experiments to show the scalability and adaptability of our algorithm.

show abstract

“…Being able to analyze data as it streams rather than storing and using data mining techniques offers the promise of more timely analysis as well as allowing more data to be processed with fewer resources. In this paper, we describe some of the autonomic self-healing capabilities of a stream processing system we are prototyping called System-S [1,14,10].…”

Section: Introductionmentioning

confidence: 99%

Towards Autonomic Fault Recovery in System-S

Jacques-Silva

Challenger

Degenaro

et al. 2007

Fourth International Conference on Autonomic Computing (ICAC'07)

View full text Add to dashboard Cite

System-S is a stream processing infrastructure which enables program fragments to be distributed and connected to form complex applications. There may be potentially tens of thousands of interdependent and heterogeneous program fragments running across thousands of nodes. While the scale and interconnection imply the need for automation to manage the program fragments, the need is intensified because the applications operate on live streaming data and thus need to be highly available. System-S has been designed with components that autonomically manage the program fragments, but the system components themselves are also susceptible to failures which can jeopardize the system and its applications.The work we present addresses the self healing nature of these management components in System-S. In particular, we show how one key component of System-S, the job management orchestrator, can be abruptly terminated and then recover without interrupting any of the running program fragments by reconciling with other autonomous system components. We also describe techniques that we have developed to validate that the system is able to autonomically respond to a wide variety of error conditions including the abrupt termination and recovery of key system components. Finally, we show the performance of the job management orchestrator recovery for a variety of workloads.

show abstract

Adaptive Control of Extreme-scale Stream Processing Systems

Cited by 109 publications

References 10 publications

Streaming the Web: Reasoning over dynamic data

Streaming the Web: Reasoning over dynamic data

Biologically-Inspired Distributed Middleware Management for Stream Processing Systems

Towards Autonomic Fault Recovery in System-S

Contact Info

Product

Resources

About