Formal Analysis of a Distributed Algorithm for Tracking Progress

Abadi, Martı́n; McSherry, Frank; Murray, Derek G.; Rodeheffer, Thomas L.

doi:10.1007/978-3-642-38592-6_2

Cited by 7 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ISpec implies that always, for all p ∈ P , there exists a local history H(p) for p such that LocState(p) = Π Loc g(H(p)) and NotRequests(p) = Π NR g(H(p)). Previous work [4] studies a distributed algorithm for tracking the progress of a computation, and arrives at a somewhat analogous result. This previous work assumes a notion of virtual time but defines neither a dataflow model nor a corresponding could-result-in relation (so, in particular, it does not treat analogues of Conditions 1 and 2).…”

Section: An Applicationmentioning

confidence: 90%

“…The development and presentation of timely dataflow in the context of Naiad was fairly precise but informal. Only one of its critical components (a distributed algorithm that keeps track of virtual times for which there may remain work) was rigorously specified and verified [4]. Moreover, in the context of Naiad, definitions focus on particular structures of dataflow graphs and particular types of nodes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Timely Dataflow: A Model

Abadi

Isard

2015

Formal Techniques for Distributed Objects, Components, and Systems

Self Cite

View full text Add to dashboard Cite

show abstract

Section: An Applicationmentioning

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Timely Dataflow: A Model

Abadi

Isard

2015

Formal Techniques for Distributed Objects, Components, and Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…When a worker receives a progress update (p, δ ), it adds δ to the local occurrence count for p. This protocol has an important safety property: no local frontier ever moves ahead of the global frontier, taken across all outstanding events in the system. Therefore, if some worker has a pending notification at p = (t, v) and p is in the local frontier, p must also be in the global frontier and the worker can safely deliver the notification to v. A formal specification of the protocol and a safety proof are presented in a separate paper [4].…”

Section: Distributed Progress Trackingmentioning

confidence: 99%

Naiad

Murray

McSherry

Isaacs

et al. 2013

Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles

Self Cite

508

View full text Add to dashboard Cite

Naiad is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream processors, and the ability to perform iterative and incremental computations. Although existing systems offer some of these features, applications that require all three have relied on multiple platforms, at the expense of efficiency, maintainability, and simplicity. Naiad resolves the complexities of combining these features in one framework.A new computational model, timely dataflow, underlies Naiad and captures opportunities for parallelism across a wide class of algorithms. This model enriches dataflow computation with timestamps that represent logical points in the computation and provide the basis for an efficient, lightweight coordination mechanism.We show that many powerful high-level programming models can be built on Naiad's low-level primitives, enabling such diverse tasks as streaming data analysis, iterative machine learning, and interactive graph mining. Naiad outperforms specialized systems in their target application domains, and its unique features enable the development of new high-performance applications.

show abstract

“…This enables very fine-grained coordination, without system-level intervention. Further technical details of progress tracking in timely dataflows can be found in [22,4].…”

Section: Timely Dataflow Conceptsmentioning

confidence: 99%

Megaphone: Latency-conscious state migration for distributed streaming dataflows

Hoffmann,

Lattuada,

McSherry

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

We design and implement Megaphone, a data migration mechanism for stateful distributed dataflow engines with latency objectives. When compared to existing migration mechanisms, Megaphone has the following differentiating characteristics: (i) migrations can be subdivided to a configurable granularity to avoid latency spikes, and (ii) migrations can be prepared ahead of time to avoid runtime coordination. Megaphone is implemented as a library on an unmodified timely dataflow implementation, and provides an operator interface compatible with its existing APIs. We evaluate Megaphone on established benchmarks with varying amounts of state and observe that compared to naïve approaches Megaphone reduces service latencies during reconfiguration by orders of magnitude without significantly increasing steady-state overhead.

show abstract

Formal Analysis of a Distributed Algorithm for Tracking Progress

Cited by 7 publications

References 8 publications

Timely Dataflow: A Model

Timely Dataflow: A Model

Naiad

Megaphone: Latency-conscious state migration for distributed streaming dataflows

Contact Info

Product

Resources

About