State-machine replication for planet-scale systems

Abstract

Online applications now routinely replicate their data at multiple sites around the world. In this paper we present Atlas, the first state-machine replication protocol tailored for such planet-scale systems. Atlas does not rely on a distinguished leader, so clients enjoy the same quality of service independently of their geographical locations. Furthermore, clientperceived latency improves as we add sites closer to clients. To achieve this, Atlas minimizes the size of its quorums using an observation that concurrent data center failures are rare. It also processes a high percentage of accesses in a single round trip, even when these conflict. We experimentally demonstrate that Atlas consistently outperforms state-of-the-art protocols in planet-scale scenarios. In particular, Atlas is up to two times faster than Flexible Paxos with identical failure assumptions, and more than doubles the performance of Egalitarian Paxos in the YCSB benchmark.

Efficient replication via timestamp stability

“…The protocol saturates at around 4K clients per site, when the outgoing network bandwidth at the leader reaches 95% usage. The fact that the leader can be a bottleneck in leader-based protocol has been reported by several prior works [13,23,24,31,43].…”

“…Unfortunately, all existing leaderless SMR protocols suffer from drawbacks in the way they order commands. Some protocols [1,5,13,31] maintain explicit dependencies between commands: a replica may execute a command only after all its dependencies get executed. These dependencies may form arbitrary long chains.…”

Scalable but wasteful

“…Given the pervasiveness of replicated state machines in distributed systems and applications, improving the replication throughput has been an important problem in the past decade. Some solutions [9,23] call for boosting throughput and reducing latency at the same time, while others [7,35] argue for trading off some latency in exchange for better throughput. Both camps, however, take a similar high-level approach for improving their performance.…”

“…Usually, the bottleneck is at the leader, as it is used to communicate with the clients and coordinate the replication [1]. Systems like EPaxos [23,31] and Atlas [9] avoid having a one-node bottleneck by not having a single leader that centers all communication around it. Instead, these systems expand on Fast Paxos [17] ideas and try to use fast quorums to commit/replicate operations in one round-trip network latency from any node in the system.…”

“…For example, many traditional protocols [2,17,24,32] rely on a dedicated leader to prescribe the operations and their order to the follower nodes. EPaxos [23] conjectures that a single leader is a bottleneck for both the throughput and latency and removes it by allowing any node to become an opportunistic leader; other recent systems [9,31] further optimize EPaxos. Pig-Paxos [7] moves a significant portion of communication from the leader onto the followers.…”