Improving Resource Efficiency at Scale with Heracles

Abstract

User-facing, latency-sensitive services, such as websearch, underutilize their computing resources during daily periods of low traffic. Reusing those resources for other tasks is rarely done in production services since the contention for shared resources can cause latency spikes that violate the service-level objectives of latency-sensitive tasks. The resulting under-utilization hurts both the affordability and energy efficiency of large-scale datacenters. With the slowdown in technology scaling caused by the sunsetting of Moore's law, it becomes important to address this opportunity. We present Heracles, a feedback-based controller that enables the safe colocation of best-effort tasks alongside a latency-critical service. Heracles dynamically manages multiple hardware and software isolation mechanisms, such as CPU, memory, and network isolation, to ensure that the latency-sensitive job meets latency targets while maximizing the resources given to best-effort tasks. We evaluate Heracles using production latency-critical and batch workloads from Google and demonstrate average server utilizations of 90% without latency violations across all the load and colocation scenarios that we evaluated. CCS Concepts: r Computer systems organization → Cloud computing; r Software and its engineering → Scheduling

Tailbench: a benchmark suite and evaluation methodology for latency-critical applications

“…Servers running latency-critical applications operate at low utilization to guard against queuing delays, long requests, and other sources of performance variability. Further, their spare capacity cannot be used by batch applications, as uncontrolled sharing of cores, caches, and power causes high and unpredictable tail latency degradation [30,33,36]. As a result, datacenters servers typically have utilizations of 5-30% [8,9,37].…”

“…These techniques include new cluster managers that schedule and migrate applications across systems to reduce interference [18,32,36,54], fast dynamic voltage-frequency scaling (DVFS) techniques to improve power efficiency [25,29,32,48], hardware and software schemes to use low power idle states [37,39,53], and hardware resource partitioning schemes that allow batch workloads to run alongside latency-critical ones, improving utilization [29,30,33,57].…”

“…Many of these studies use workloads internal to datacenter operators like Google or Facebook [32,33,36,38,55,56]. Academic studies use one or a few latencycritical benchmarks [25,48,54], which limits the range of behaviors and performance requirements across which their proposed techniques can be evaluated.…”

“…Latency-critical applications have a wide variety of latency requirements and microarchitectural characteristics. However, most recent work in this area uses one or a few latencycritical applications in their evaluations [25,32,33,48], which do not stress a wide range of behaviors. Some prior work in this area even uses more readily-available sequential and parallel batch workloads (e.g., from SPEC CPU2006 or PAR-SEC [12]) and treats them as latency-critical applications [15,57].…”

Energy proportionality and workload consolidation for latency-critical applications

“…This work is orthogonal to ours and could be a useful additional signal for our control plane. Heracles manages multiple hardware and software isolation mechanisms, including packet scheduling and cache partitioning, to co-locate latency-sensitive applications with batch tasks while maintaining millisecond SLOs [30]. We limit our focus to DVFS and core assignment but target more aggressive SLOs.…”