Learning-based Memory Allocation for C++ Server Workloads

Abstract

Modern C++ servers have memory footprints that vary widely over time, causing persistent heap fragmentation of up to 2× from long-lived objects allocated during peak memory usage. This fragmentation is exacerbated by the use of huge (2MB) pages, a requirement for high performance on large heap sizes. Reducing fragmentation automatically is challenging because C++ memory managers cannot move objects.This paper presents a new approach to huge page fragmentation. It combines modern machine learning techniques with a novel memory manager (Llama) that manages the heap based on object lifetimes and huge pages (divided into blocks and lines). A neural network-based language model predicts lifetime classes using symbolized calling contexts. The model learns context-sensitive per-allocation site lifetimes from previous runs, generalizes over different binary versions, and extrapolates from samples to unobserved calling contexts. Instead of size classes, Llama's heap is organized by lifetime classes that are dynamically adjusted based on observed behavior at a block granularity.Llama reduces memory fragmentation by up to 78% while only using huge pages on several production servers. We address ML-specific questions such as tolerating mispredictions and amortizing expensive predictions across application execution. Although our results focus on memory allocation, the questions we identify apply to other system-level problems with strict latency and resource requirements where machine learning could be applied.

Ofc

“…Machine learning for resource management. A number of works have leveraged ML to optimize server applications and cloud infrastructures [2,10,11,16,18,25]. We focus here on the most closely related to our work.…”

A Machine Learning Framework to Improve Storage System Performance

“…But the performance improvements that they reported were negligible (0.1-6%). Nevertheless, there is a growing trend in using ML techniques to solve storage and OS problems: predicting index structures in key-value stores [17,38], memory allocation [47], TCP congestion control [24], offline black-box optimization for storage parameters [8], database query optimization [37], local and distributed caching [60,66] and cloud resource management [16,19,20].…”

Toward reconfigurable kernel datapaths with learned optimizations

“…Machine learning techniques have produced early but successful results in computer systems, replacing well-tuned index structures for data retrieval [29], predicting hardware device state for better management [24], and managing C++ object memory efficiently [39]. Zhang and Huang [54] have argued that ML should be applied to the OS kernel as well.…”

“…ML for systems. ML has found applications in index retrieval [29], bloom filter queries [29], CPU scheduling [14], C++ memory management [39], and many other contexts. Zhang and Huang [54] argue for its use to optimize OS kernels.…”