Mergeable replicated data types

Abstract

Programming geo-replicated distributed systems is challenging given the complexity of reasoning about different evolving states on different replicas. Existing approaches to this problem impose significant burden on application developers to consider the effect of how operations performed on one replica are witnessed and applied on others. To alleviate these challenges, we present a fundamentally different approach to programming in the presence of replicated state. Our insight is based on the use of invertible relational specifications of an inductively-defined data type as a mechanism to capture salient aspects of the data type relevant to how its different instances can be safely merged in a replicated environment. Importantly, because these specifications only address a data type's (static) structural properties, their formulation does not require exposing low-level system-level details concerning asynchrony, replication, visibility, etc. As a consequence, our framework enables the correct-by-construction synthesis of rich merge functions over arbitrarily complex (i.e., composable) data types. We show that the use of a rich relational specification language allows us to extract sufficient conditions to automatically derive merge functions that have meaningful non-trivial convergence properties. We incorporate these ideas in a tool called Quark, and demonstrate its utility via a detailed evaluation study on real-world benchmarks. CCS Concepts: • Computing methodologies → Distributed programming languages; • Computer systems organization → Availability; • Software and its engineering → Formal software verification.

ECROs: building global scale systems from sequential code

“…Cloud Types [Burckhardt et al 2012] do not impose restrictions on operations but require user-defined merge functions. Mergeable Replicated Data Types [Kaki et al 2019] use a three-way merge function and invertible relational specifications to derive correct merge functions. However, the aforementioned RDTs do not consider application invariants.…”

“…To ease the development of geo-distributed applications, much work has studied the concept of a replicated data type (RDT) [Burckhardt et al 2012;Kaki et al 2019;Shapiro et al 2011b]. The goal of an RDT is to expose the same interface as its sequential counterpart and embed in its implementation mechanisms to enforce correctness.…”

“…However, devising correct merge procedures is a difficult task. Mergeable replicated data types [Kaki et al 2019] automatically derive merge procedures from invertible relational specifications that convert the RDT's state to relations over sets and back. However, the merge semantics can be hard to customize as they depend on the set relations being used.…”

Composing and decomposing op-based CRDTs with semidirect products

“…Mergeable Replicated Data Types (MRDTs), defined by Kaki et al [2019] and built on top of Irmin [MirageOS Project 2020], are branch-and-merge based replicated data types that use a three-way merge function for sets to define MRDTs for various data types automatically. Like us, Kaki et al define a replicated integer register supporting addition and multiplication operations, but with different semantics: a multiplication is treated as its equivalent addition.…”

From causality to stability: understanding and reducing meta-data in CRDTs

“…To ease the development of geo-distributed applications, much research has studied the concept of replicated data types which expose to programmers an interface akin to that of a sequential data type while implementing mechanisms to keep data consistent across replicas [7,12,16]. Conflict-Free Replicated Data Types [14][15][16] (CRDTs) are replicated data structures that can be concurrently updated without requiring synchronization among replicas to deal with conflicts.…”