The dataflow model

Abstract

Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business (e.g. Web logs, mobile usage statistics, and sensor networks). At the same time, consumers of these datasets have evolved sophisticated requirements, such as event-time ordering and windowing by features of the data themselves, in addition to an insatiable hunger for faster answers. Meanwhile, practicality dictates that one can never fully optimize along all dimensions of correctness, latency, and cost for these types of input. As a result, data processing practitioners are left with the quandary of how to reconcile the tensions between these seemingly competing propositions, often resulting in disparate implementations and systems. We propose that a fundamental shift of approach is necessary to deal with these evolved requirements in modern data processing. We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete, and instead live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable is via principled abstractions that allow the practitioner the choice of appropriate tradeoffs along the axes of interest: correctness, latency, and cost. In this paper, we present one such approach, the Dataflow Model, along with a detailed examination of the semantics it enables, an overview of the core principles that guided its design, and a validation of the model itself via the real-world experiences that led to its development.

One SQL to Rule Them All - an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables

“…Following the Millwheel model [4], Flink supports record timestamps and watermarks to decide when a computation can be performed. 5 Note that these semantics are different from those for the STREAM keyword proposed in 6.5.1.…”

“…Apache Beam [15] has recently added SQL support, developed with a careful eye towards Beam's unification of bounded and unbounded data processing [5]. Beam currently implements a subset of the semantics proposed by this paper, and many of the proposed extensions have been informed by our experiences with Beam over the years.…”

“…Millwheel [4] based its processing instead on watermarks, directly computing on the out-of-order data along with metadata about how complete the input was believed to be. This approach was further extended in Google's Cloud Dataflow [5], which pioneered the out-of-order processing model adopted in both Beam and Flink.…”

Harnessing sliding-window execution semantics for parallel stream processing

“…The timebased model is another example of FCF windows which are often easier to understand for the final users, as they can relate the windows evolution directly to time (this is useful for business applications like in e-commerce and on-line auction systems). The use of session windows [21] emphasizes the variability of the window extents that are defined based on the frequency of input tuples. Session windows are disjoint (not-overlapped) and their extents consist in a set of consecutive input tuples whose time gap (between two consecutive tuples) does not exceed a defined inactivity gap parameter.…”

“…Session windows are disjoint (not-overlapped) and their extents consist in a set of consecutive input tuples whose time gap (between two consecutive tuples) does not exceed a defined inactivity gap parameter. They are used when the input stream is highly irregularly distributed over the time and are useful to model the clients' behavior like in network monitoring applications [21]. Hybrid models like slide-by-tuple windows and the dual slide-by-time (i.e.…”