Implementation of Cluster-wide Logical Clock and Causal Consistency in MongoDB

Abstract

MongoDB is a distributed database that supports replication and horizontal partitioning (sharding). MongoDB replica sets consist of a primary that accepts all client writes and then propagates those writes to the secondaries. Each member of the replica set contains the same set of data. For horizontal partitioning, each shard (or partition) is a replica set. This paper discusses the design and rationale behind MongoDB's implementation of a cluster-wide logical clock and causal consistency. The design leveraged ideas from across the research community to ensure that the implementation adds minimal processing overhead, tolerates possible operator errors, and gives protection against non-trusted client attacks. While the goal of the team was not to discover or test new algorithms, the practical implementation necessitated a novel combination of ideas from the research community on causal consistency, security, and minimal performance overhead at scale. This paper describes a large scale, practical implementation of causal consistency using a hybrid logical clock, adding the signing of logical time ranges to the protocol, and introducing performance optimizations necessary for systems at scale. The implementation seeks to define an event as a state change and as such must make forward progress guarantees even during periods of no state changes for a partition of data.

CockroachDB: The Resilient Geo-Distributed SQL Database

“…There has been a great deal of work both in industry and in the literature to support distributed transactions with varying levels of consistency and scalability. Over the years, many systems with reduced consistency levels have been proposed with the goal of overcoming the scalability challenges of traditional relational database systems [5,15,19,32,35,44,61,64,71]. For many applications, however, isolation levels below serializable permit dangerous anomalies, which may manifest as security vulnerabilities [73].…”

Adapting TPC-C benchmark to measure performance of multi-document transactions in MongoDB

“…[7] A single snapshot of the data is used for the duration of the transaction. A snapshot is a single point in time view of the data at a distinct cluster time maintained via a cluster-wide logical clock [34]. Once a transaction begins with a snapshot at cluster time, no subsequent writes outside of that transaction's context occurring after that cluster time will be seen within the transaction.…”

Distributed causal memory: modular specification and verification in higher-order distributed separation logic

“…The CAP theorem [Gilbert and Lynch 2002] states that a distributed database cannot at the same time provide consistency, availability, and partition (failure) tolerance. Hence, many such systems choose to sacrifice aspects of data consistency for the sake of availability and fault tolerance [Bailis et al 2013;Chang et al 2008;Lloyd et al 2011;Tyulenev et al 2019]. In those systems different replicas of the database may, at the same point in time, observe different, inconsistent data.…”