P axos made transparent

Abstract

State machine replication (SMR) leverages distributed consensus protocols such as PAXOS to keep multiple replicas of a program consistent in face of replica failures or network partitions. This fault tolerance is enticing on implementing a principled SMR system that replicates general programs, especially server programs that demand high availability. Unfortunately, SMR assumes deterministic execution, but most server programs are multithreaded and thus nondeterministic. Moreover, existing SMR systems provide narrow state machine interfaces to suit specific programs, and it can be quite strenuous and error-prone to orchestrate a general program into these interfaces This paper presents CRANE, an SMR system that transparently replicates general server programs. CRANE achieves distributed consensus on the socket API, a common interface to almost all server programs. It leverages deterministic multithreading (specifically, our prior system PARROT) to make multithreaded replicas deterministic. It uses a new technique we call time bubbling to efficiently tackle a difficult challenge of nondeterministic network input timing. Evaluation on five widely used server programs (e.g., Apache, ClamAV, and MySQL) shows that CRANE is easy to use, has moderate overhead, and is robust. CRANE's source code is at github.com/columbia/crane.

Proceedings of the 15th ACM International Conference on Computing Frontiers

“…RepFrame [25] enables running incompatible stock analyses, based on state machine replication via the Paxos consensus protocol at the level of the POSIX socket API [24]. While RepFrame does not require rewrite rules, it puts important restrictions both on the types of applications that it can run (i.e.…”

Recent Results on Fault-Tolerant Consensus in Message-Passing Networks

“…ZooKeeper is later used in many practical storage systems, like HBase [5] and Salus [139]. Recently, many novel mechanisms have been proposed to improve the performance of Paxos, including quorum lease [96], diskless Paxos [127], even load balancing [95], and time bubbling (for handling nondeterministic network input timing) [50]. While the original Paxos [78,79] is theoretically elegant, practitioners have found it hard to implement Paxos in practice [39].…”

Apus

“…The strong fault-tolerance of SMR makes it an ideal high-availability service for general server programs. Recent SMR systems [29,40,49] use Paxos to enforce the same inputs for a server program, and they use advanced techniques (e.g., deterministic inter-thread synchronization [29,78]) to make the program transit the same execution states across replicas. These SMR systems tolerate hardware failures for server programs.…”

“…Unfortunately, despite much effort, state-of-the-art still lacks a fast, scalable Paxos protocol for general server programs. A main reason is that traditional Paxos protocols [29,66,75] go through software network layers in OS kernels [72], which incurs high consensus latency. For efficiency, Paxos protocols typically take the Multi-Paxos approach [54]: it assigns one replica as the "leader" to invoke consensus requests, and the other replicas as "backups" to agree on requests.…”