Tales of the Tail

Abstract

Interactive services often have large-scale parallel implementations. To deliver fast responses, the median and tail latencies of a service's components must be low. In this paper, we explore the hardware, OS, and application-level sources of poor tail latency in high throughput servers executing on multi-core machines.We model these network services as a queuing system in order to establish the best-achievable latency distribution. Using fine-grained measurements of three different servers (a null RPC service, Memcached, and Nginx) on Linux, we then explore why these servers exhibit significantly worse tail latencies than queuing models alone predict. The underlying causes include interference from background processes, request re-ordering caused by poor scheduling or constrained concurrency models, suboptimal interrupt routing, CPU power saving mechanisms, and NUMA effects.We systematically eliminate these factors and show that Memcached can achieve a median latency of 11 µs and a 99.9th percentile latency of 32 µs at 80% utilization on a four-core system. In comparison, a naïve deployment of Memcached at the same utilization on a single-core system has a median latency of 100 µs and a 99.9th percentile latency of 5 ms. Finally, we demonstrate that tradeoffs exist between throughput, energy, and tail latency.

Hipster: Hybrid Task Manager for Latency-Critical Cloud Workloads

“…The statemachine configuration for Memcached and Web-Search are represented in blue and red line, respectively, in Figure 2c. Figure 3 [30] and a QoS target that needs to be met. As shown in Figure 2 and 3, there exists a unique configuration for each load that optimizes energy efficiency.…”

The Hipster Approach for Improving Cloud System Efficiency

“…In practical scenarios, each workload has a time-varying load [38] and a QoS target that needs to be met. As shown in Figure 2 and 3, there exists a unique con guration for each load that optimizes energy e ciency.…”

“…At runtime, Hipster determines when to dynamically switch between the learning and exploitation phases, based on a pre xed time quantum. At deployment stage, we ensure that the bucket size for each workload gives at least 95 % QoS guarantee [38] with minimal energy consumption.…”

The IX Operating System

“…Ideally, each service node will use the fewest resources (cores, memory, or IOPS) needed to satisfy packet rate and tail latency requirements at any point. Unfortunately, classic operating system schedulers are illmatched to ensure tail control [Leverich and Kozyrakis 2014;Li et al 2014]. Novel dynamic resource management mechanisms and policies are required to improve energy proportionality and workload consolidation in the presence of latency-sensitive applications [Lo et al 2014[Lo et al , 2015.…”

“…They are particularly sensitive to resource allocation and frequency settings, and they suffer frequent tail latency violations when common power management or consolidation approaches are used [Leverich and Kozyrakis 2014;Li et al 2014]. As a result, operators typically deploy them on dedicated servers running in polling mode, forgoing opportunities for workload consolidation and reduced power consumption at below-peak utilization levels.…”