Hardware Translation Coherence for Virtualized Systems

Abstract

To improve system performance, modern operating systems (OSes) often undertake activities that require modification of virtual-to-physical page translation mappings. For example, the OS may migrate data between physical frames to defragment memory and enable superpages. The OS may migrate pages of data between heterogeneous memory devices. We refer to all such activities as page remappings. Unfortunately, page remappings are expensive. We show that translation coherence is a major culprit and that systems employing virtualization are especially badly affected by their overheads. In response, we propose hardware translation invalidation and coherence or HATRIC, a readily implementable hardware mechanism to piggyback translation coherence atop existing cache coherence protocols. We perform detailed studies using KVM-based virtualization, showing that HATRIC achieves up to 30% performance and 10% energy benefits, for per-CPU area overheads of 2%. We also quantify HATRIC's benefits on systems running Xen and find up to 33% performance improvements.

Attc (@C)

“…As a result, in addition to the list of victim vCPUs approximated by the guest OS, the list of victim cores is approximated by the VMM. Combined with the flushing of translation structures upon a nested page table update, these approximations result in frequent needless evictions of unrelated translations [44]. TLB shootdown activity has been observed to be a significant bottleneck in prior studies [4,6,12,25,33,40,44] and is also confirmed by our own experiments.…”

“…Current VMMs do not track the gVA of pages used by the guest. Since modern processors only permit invalidations of individual TLB entries when the gVA (for guest pages) is known, when the VMM updates the nested page table, translation structures are completely flushed [44]. Repopulating the flushed 2-dimensional page tables are very expensive.…”

ECO TLB

“…Hardware-based TLB shootdown. There have been a number of approaches to handle the problem of TLB cache coherence at the hardware layer [7,10,12,42,43,48,49,51,60,62]. Several of these hardware-based approaches attempt to squeeze performance using non-traditional TLB designs, such as multi-level TLB hierarchies.…”

“…Furthermore, UNITD adds a costly content-addressable memory (CAM) to each TLB to perform reverse address translations when checking whether a page translation is present in a specific TLB, thereby greatly increasing the TLB's power consumption. HATRIC [62] is a hardware mechanism similar to UNITD and piggybacks translation coherence information using the existing cache coherence protocols.…”

“…To address the synchronous TLB shootdown overhead, hardware-based approaches [7,42,48,49,51,60,62] strive to provide TLB coherence in an efficient manner. Nevertheless, such hardware mechanisms need expensive hardware modification and are not available as part of the existing x86 architecture.…”