Cheetah: A High-Speed Programmable Load-Balancer Framework With Guaranteed Per-Connection-Consistency

Abstract

Large service providers use load balancers to dispatch millions of incoming connections per second towards thousands of servers. There are two basic yet critical requirements for a load balancer: uniform load distribution of the incoming connections across the servers, which requires to support advanced load balancing mechanisms, and per-connection-consistency (PCC), i.e, the ability to map packets belonging to the same connection to the same server even in the presence of changes in the number of active servers and load balancers. Yet, simultaneously meeting these requirements has been an elusive goal. Today's load balancers minimize PCC violations at the price of non-uniform load distribution. This paper presents CHEETAH, a load balancer that supports advanced load balancing mechanisms and PCC while being scalable, memory efficient, fast at processing packets, and offers comparable resilience to clogging attacks as with today's load balancers. The CHEETAH LB design guarantees PCC for any realizable server selection load balancing mechanism and can be deployed in both stateless and stateful manners, depending on operational needs. We implemented CHEETAH on both a software and a Tofino-based hardware switch. Our evaluation shows that a stateless version of CHEETAH guarantees PCC, has negligible packet processing overheads, and can support load balancing mechanisms that reduce the flow completion time by a factor of 2 − 3×.

Impact of Distributed Rate Limiting on Load Distribution in a Latency-sensitive Messaging Service

“…The objective is to evenly distribute an aggregate load across servers to optimize performance given an available set of servers. Loadbalancers [21,14,26,37] can therefore be effective in minimizing processing latency. However, as illustrated in Fig.…”

“…The modification is that in the token bucket system, messages experience a delay if and only if the queue content in the G/D/1 system exceeds b − 1. In other words, assuming a first-served-first-come (FCFS) order of service 14 , the token bucket delay d i of the i th message can be obtained from the system time of this message in the corresponding G/D/1 system as follows:…”

“…Note that the sum of message delays divided by the number of messages gives the ensemble mean message latency (up to time t) 14. Note that a FCFS service order is known to minimize the sum of message latencies in both single-server and (homogeneous) multi-server systems when service times are constant[46].…”

Charon: Load-Aware Load-Balancing in P4

“…They encode server id information in packet headers and daisychain two possible server candidates, to retrieve a potentially changed flow-server mappings. Charon adopts stateless load balancing scheme [9], [10], [12] and encapsulates the server id inside the packet. In particular, the TCP timestamp option [13] is used to transport this information.…”

“…Different covert channels are available (e.g. connection-id of QUIC connections and the least significant bits of IPv6 addresses) [12]. This paper uses the higher-bits of TCP timestamp options.…”