Efficient, Consistent Distributed Computation with Predictive Treaties

Magrino, Tom; Liu, Jed; Foster, Nate; Gehrke, Johannes; Myers, Andrew C.

doi:10.1145/3302424.3303987

Cited by 4 publications

(2 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thereafter, it carefully adjusts the application data rate to match the available bandwidth, while maximizing the achievable accuracy. Besides, Magrino et al introduce predictive treaties to predict the evolution of the system state in distributed transaction processing [34]. This method can reduce the coordination of geo-distributed applications and improve their performance.…”

Section: Related Workmentioning

confidence: 99%

SDTP: Accelerating Wide-Area Data Analytics With Simultaneous Data Transfer and Processing

Chen

Luo

Rottenstreich

et al. 2023

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

For the efficient analysis of geo-distributed datasets, cloud providers implement data-parallel jobs across geo-distributed sites (e.g., datacenters and edge clusters), which are generally interconnected by wide-area network links. However, current state-ofthe-art geo-distributed data analytic methods fail to make full use of the available network and computing resources. The main reason is that such geo-distributed methods must wait for bottleneck sites to complete the corresponding transmission and computation in each phase. Furthermore, such geo-distributed methods may be impractical to the network bandwidth dynamicity and diverse job parallelism. To this end, we propose a Simultaneous Data Transfer and Processing (SDTP) mechanism to accelerate wide-area data analytics, with the joint consideration of network bandwidth dynamics and job parallelism. In the SDTP, a site can execute the computation, provided that it obtains the required input data. As a result, the input data loading, map, shuffle, and reduce phases at each site need not wait for the completion of the previous phases of other sites. We further improve the SDTP method by offering more accurate time estimation and generalizing the mechanism to dynamic situations. The trace-driven results demonstrate that SDTP can improve the wide-area analytic job response time by 19% to 72% compared to other methods.

show abstract

Section: Related Workmentioning

confidence: 99%

SDTP: Accelerating Wide-Area Data Analytics With Simultaneous Data Transfer and Processing

Chen

Luo

Rottenstreich

et al. 2023

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

show abstract

“…The second type of consistency annotation we allow is application-centric, and makes use of Hydrologic's declarative formulation. Past work has demonstrated that invariants are a powerful way for developers to precisely specify what guarantees are necessary at application level [18,21,30,45,60,74,79,84]. These include motonicity invariants that guarantee convergent outcomes, or isolation invariants for predicates on visible states-e.g., positive bank accounts or referential integrity.…”

Section: Design Spacementioning

confidence: 99%

New Directions in Cloud Programming

Cheung¹,

Crooks²,

Hellerstein³

et al. 2021

Preprint

View full text Add to dashboard Cite

Nearly twenty years after the launch of AWS, it remains difficult for most developers to harness the enormous potential of the cloud. In this paper we lay out an agenda for a new generation of cloud programming research aimed at bringing research ideas to programmers in an evolutionary fashion. Key to our approach is a separation of distributed programs into a PACT of four facets: Program semantics, Availablity, Consistency and Targets of optimization. We propose to migrate developers gradually to PACT programming by lifting familiar code into our more declarative level of abstraction. We then propose a multi-stage compiler that emits humanreadable code at each stage that can be hand-tuned by developers seeking more control. Our agenda raises numerous research challenges across multiple areas including language design, query optimization, transactions, distributed consistency, compilers and program synthesis.

show abstract