Rollback-Recovery for Middleboxes

Abstract

Network middleboxes must offer high availability, with automatic failover when a device fails. Achieving high availability is challenging because failover must correctly restore lost state (e.g., activity logs, port mappings) but must do so quickly (e.g., in less than typical transport timeout values to minimize disruption to applications) and with little overhead to failure-free operation (e.g., additional per-packet latencies of 10-100s of µs). No existing middlebox design provides failover that is correct, fast to recover, and imposes little increased latency on failure-free operations. We present a new design for fault-tolerance in middleboxes that achieves these three goals. Our system, FTMB (for Fault-Tolerant MiddleBox), adopts the classical approach of "rollback recovery" in which a system uses information logged during normal operation to correctly reconstruct state after a failure. However, traditional rollback recovery cannot maintain high throughput given the frequent output rate of middleboxes. Hence, we design a novel solution to record middlebox state which relies on two mechanisms: (1) 'ordered logging', which provides lightweight logging of the information needed after recovery, and (2) a 'parallel release' algorithm which, when coupled with ordered logging, ensures that recovery is always correct. We implement ordered logging and parallel release in Click and show that for our test applications our design adds only 30µs of latency to median per packet latencies. Our system introduces moderate throughput overheads (5-30%) and can reconstruct lost state in 40-275ms for practical systems. CCS Concepts • Networks → Middleboxes / network appliances; • Computer systems organization → Availability;

Towards Practical Default-On Multi-Core Record/Replay

“…On SPLASH-2, Chimera saw overhead ranging from 1.6× for two cores to over 3× for 8 cores. Similar systems exhibited substantially larger overheads [24] or only considered small and easily parallelizable programs [43].…”

Towards Carrier-Grade Service Provisioning in NFV

“…One or more backup nodes are assigned to a flow (constraints (11)). A flow is assigned a backup node provided that it is using atleast one backup instance hosted on the node (constraint (12)). This model is an Integer Non-linear Program (INLP) because of the non linearity of equation (10).…”

“…Typically, NFs need to maintain 10-100s of state variables that are per-flow or shared across flows [10]. Backup instances of stateful NFs need to have updated state information to ensure successful failover and service continuity [11], [12].…”

Research Challenges for Network Function Virtualization - Re-Architecting Middlebox for High Performance and Efficient, Elastic and Resilient Platform to Create New Services -

“…Active-active replication, where master and slave are executed on all inputs but only the master's output is released to users, will not work because of the non-deterministic nature of packet processing in middleboxes. Sherry et al [130], [131] proposed a fault-tolerant middlebox, a new design for fault-tolerant middleboxes that achieves correctness, fast recovery with only a slight increase in latency. They took a replay-based approach that maintains a log of inputs to the system and recreates the lost state by replaying the inputs from the log in the event of a failure.…”