Hermes: A Fast, Fault-Tolerant and Linearizable Replication Protocol

Abstract

Today's datacenter applications are underpinned by datastores that are responsible for providing availability, consistency, and performance. For high availability in the presence of failures, these datastores replicate data across several nodes. This is accomplished with the help of a reliable replication protocol that is responsible for maintaining the replicas strongly-consistent even when faults occur. Strong consistency is preferred to weaker consistency models that cannot guarantee an intuitive behavior for the clients. Furthermore, to accommodate high demand at real-time latencies, datastores must deliver high throughput and low latency.This work introduces Hermes 1 , a broadcast-based reliable replication protocol for in-memory datastores that provides both high throughput and low latency by enabling local reads and fully-concurrent fast writes at all replicas. Hermes couples logical timestamps with cache-coherence-inspired invalidations to guarantee linearizability, avoid write serialization at a centralized ordering point, resolve write conflicts locally at each replica (hence ensuring that writes never abort) and provide fault-tolerance via replayable writes. Our implementation of Hermes over an RDMA-enabled reliable datastore with five replicas shows that Hermes consistently achieves higher throughput than state-of-the-art RDMA-based reliable protocols (ZAB and CRAQ) across all write ratios while also significantly reducing tail latency. At 5% writes, the tail latency of Hermes is 3.6× lower than that of CRAQ and ZAB.

Efficient replication via timestamp stability

“…The protocol saturates at around 4K clients per site, when the outgoing network bandwidth at the leader reaches 95% usage. The fact that the leader can be a bottleneck in leader-based protocol has been reported by several prior works [13,23,24,31,43].…”

“…In particular, many storage systems orchestrate data access using a fault-tolerant timestamping service [2,3,36,42,47], usually implemented by a leader-based SMR protocol [29,35]. As reported in prior works, the leader is a potential bottleneck and is unfair with respect to client locations [14,24,25,32,44]. To sidestep these problems, leaderless protocols order commands in a fully decentralized manner.…”

Zeus

“…Several previous systems have demonstrated such load balancers [3,5,7,51]. We implement a simple one using a distributed, replicated key-value store based on Hermes [35]. We extract a key from each request and look it up in the key-value store.…”

“…Cache coherence protocols have been used to implement hardware transactions [29]. Zeus builds on ideas in Hermes [35], which adapted concepts from cache coherence and applied them to enforce strong consistency for replicated in-memory datastores. Hermes allows for local reads and fast reliable updates to individual objects from all replicas; however, it does not support multi-object reliable transactions or ownerships.…”

Odyssey

“…To this end, we first develop a taxonomy of existing protocols, classifying them into four classes based on their operational patterns (Section 3). To understand the performance of the different classes of protocols, we carefully select ten protocols for analysis: ZAB [25], Multi-Paxos [39], CHT and multi-leader CHT [10], CRAQ [62], Derecho [26], Classic Paxos (CP) [36], All-Aboard Paxos [23], ABD [48] and Hermes [33].…”