Naiad

Abstract

Naiad is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream processors, and the ability to perform iterative and incremental computations. Although existing systems offer some of these features, applications that require all three have relied on multiple platforms, at the expense of efficiency, maintainability, and simplicity. Naiad resolves the complexities of combining these features in one framework.A new computational model, timely dataflow, underlies Naiad and captures opportunities for parallelism across a wide class of algorithms. This model enriches dataflow computation with timestamps that represent logical points in the computation and provide the basis for an efficient, lightweight coordination mechanism.We show that many powerful high-level programming models can be built on Naiad's low-level primitives, enabling such diverse tasks as streaming data analysis, iterative machine learning, and interactive graph mining. Naiad outperforms specialized systems in their target application domains, and its unique features enable the development of new high-performance applications.

Stateful dataflow multigraphs

“…HPVM [45] extends the LLVM IR by introducing hierarchical dataflow graphs for mapping to accelerators, yet still lacks a high-level view and explicit state machines that SDFGs offer. Other representations include Bamboo [73], an object-oriented dataflow model that tracks state locally through data structure mutation over the course of the program; Dryad [39] and Naiad [52], parametric graphs intended for coarsegrained distributed data-parallel applications, where Naiad extends Dryad with definition of loops in a nested context; simplified data dependency graphs for optimization of GPU applications [70]; deterministic producer/consumer graphs [15]; and other combinations of task DAGs with data movement [32]. As the SDFG provides general-purpose state machines with dataflow, all the above models can be fully represented within it, where SDFGs have the added benefit of encapsulating fine-grained data dependencies.…”

“…The checks ensure that the array is indeed transient and not used in other instances of data access nodes. To avoid recomputing subsets (which may not be feasible to compute symbolically), if the transformation operates in strict mode, it only matches two arrays of the same shape (lines [51][52][53][54][55][56]. The transformation then operates in a straightforward manner, renaming the memlets to point to the second (not removed) array (lines 66-70) and redirecting dataflow edges to that data access node (lines 73-74).…”

Scaling Ordered Stream Processing on Shared-Memory Multicores

“…Aurora [2], its distributed counterpart, Borealis [1] and STREAM [11] are some of the early prototypes of stream processing engines that make dynamic scheduling decisions. Many recent stream processing engines (NaiagraST [31], Nile [24], Naiad [33],Spark Streaming [41], Storm [8], S4 [35]) also scheduling decisions during runtime. All these systems either focus on single core or sharednothing architectures.…”

Radiator - efficient message propagation in context-aware systems

“…More recently there have been proposals for implementing DSPS on Cloud infrastructure such as Stormy [49], taking advantage of its elastic characteristics (i.e., easily adding and removing nodes from the system). Additionally, systems such as Naiad [50] combine DSPS with batch processing techniques, allowing complex incremental computations on streaming data.…”