Design principles for developing stream processing applications

Abstract

Stream processing applications are used to ingest, process, and analyze continuous data streams from heterogeneous sources of live and stored data, generating streams of output results. These applications are, in many cases, complex, large‐scale, low‐latency, and distributed in nature. In this paper, we describe the design principles and architectural underpinnings for stream processing applications. These principles are distilled from our experience in building real‐world applications both for internal use as well as with customers from several industrial and academic domains. We provide principles, guidelines, as well as appropriate implementation examples to highlight the different aspects of stream processing application design and development. Copyright © 2010 John Wiley & Sons, Ltd.

A design proto‐pattern for continuously evaluated forecasting in IBM® InfoSphere® Streams

“…If multiple hosts had been available, we would have used placement . With only one host available, the data volume was instead reduced by sampling . A sampling rate of one tuple in 60 was selected.…”

“…By disseminating known good solutions, design patterns expedite software development and increase maintainability. While development of streaming applications is greatly simplified by the Streams platform, end‐to‐end solutions remain challenging to develop, as Turaga et al explain, ‘These characteristics [of streaming applications] require a shift in the developer thought process and the engineering methods employed during the application design, development, and evolution.’ We offer our proto‐pattern to further simplify, and expedite, the development of applications in Streams for multistep forecasting over any suitable data stream. The purpose of this article is not to produce accurate forecasts (yet) but to provide a pattern, the implementation of which is easily adjustable for tuning the forecasts, by changing the parameters or implementing different forecasting methods.…”

“…Stream computing answers the growing need to process an increasing extent of connectivity between an increasing number of heterogeneous data sources . Stream computing applications continually receive signals from external sources and analyze them for actionable information .…”

Streaming algorithms for k-core decomposition

“…For applications that manage dynamic graphs, applying such algorithms for every edge insertion and removal is prohibitive in terms of performance. Furthermore, batch processing takes away the ability to react to changes quickly -one of the key benefits of stream processing [28].…”

Auto-parallelizing stateful distributed streaming applications

“…In other words, the compiler starts parallel regions as close to sources as possible, and keeps adding operator instances as long as all safety conditions are satisfied. This is motivated by the observation that in practice, more operators are selective than prolific, since streaming applications tend to reduce the data volume early to reduce overall cost [26]. Furthermore, our policy of only parallelizing operator instances with selectivity  1 ensures that the number of tuples in a region shrinks from left to right.…”