Cache-aware load balancing of data center applications

Abstract

Our deployment of cache-aware load balancing in the Google web search backend reduced cache misses by ∼0.5x, contributing to a double-digit percentage increase in the throughput of our serving clusters by relieving a bottleneck. This innovation has benefited all production workloads since 2015, serving billions of queries daily. A load balancer forwards each query to one of several identical serving replicas. The replica pulls each term's postings list into RAM from flash, either locally or over the network. Flash bandwidth is a critical bottleneck, motivating an application-directed RAM cache on each replica. Sending the same term reliably to the same replica would increase the chance it hits cache, and avoid polluting the other replicas' caches. However, most queries contain multiple terms and we have to send the whole query to one replica, so it is not possible to achieve a perfect partitioning of terms to replicas. We solve this via a voting scheme, whereby the load balancer conducts a weighted vote by the terms in each query, and sends the query to the winning replica. We develop a multi-stage scalable algorithm to learn these weights. We first construct a large-scale term-query graph from logs and apply a distributed balanced graph partitioning algorithm to cluster each term to a preferred replica. This yields a good but simplistic initial voting table, which we then iteratively refine via cache simulation to capture feedback effects.

More Recent Advances in (Hyper)Graph Partitioning

“…A similar application is to boost IO throughput in the Google search engine backend by improving cache utilization. Archer et al [15] initialize a voting table that assigns search requests, using graph partitioning on a bipartite graph where vertices are search terms and queries, and an edge exists for each term contained in a query. Subsequent simulation-and-refinement further boosts the prediction accuracy (item in cache) of the voting table.…”

Parallel Flow-Based Hypergraph Partitioning

“…In this work, we consider the connectivity metric e∈E (λ(e) − 1)ω(e) where λ(e) := {V i | e ∩ V i = ∅} denotes the number of different blocks connected by hyperedge e ∈ E and ω(e) denotes its weight. Often balanced partitioning is used as an acceleration technique for other applications, such as quantum circuit simulation [31], sharding distributed databases [15,36], load balancing (for scientific computing) [13], route planning [17,32], or boosting cache utilization in a search engine backend [7].…”

Limited Associativity Caching in the Data Plane

“…Caching is a fundamental technique for boosting systems performance. In particular, software-managed caches, aka software caches, are employed in multiple data-stores and databases [2,12,15,19,20,21,36,40,42,41], operating systems, middleware, streaming services, and is a major capability of edge computing. The common motivation behind caching is to store data closer to the application than its source and avoid recalculating queries, query plans, and temporal indices.…”