On-the-fly Page Migration and Address Reconciliation for Heterogeneous Memory Systems

Abstract

For efficient placement of data in flat-address heterogeneous memory systems consisting of fast (e.g., 3D-DRAM) and slow memories (e.g., NVM), we present a hardware-based page migration technique. Unlike epoch-based approaches that migrate heavily accessed (“hot”) pages from slow to fast memories at each epoch interval, we migrate a page immediately when it becomes hot (“on-the-fly”), using hardware in user-transparent manner and with minimal OS intervention. The management of physical addresses due to page relocation becomes cumbersome and requires costly OS intervention. We use a small hardware remap table to keep track of new physical addresses of the migrated pages. This limits address reconciliation to occur only at periodic evictions of old remap entries. Also, we propose a hardware-orchestrated light-weight address reconciliation process. For our studied heterogeneous memory system, on-the-fly page migration with hardware-assisted address reconciliation provides 74% and 24% IPC improvements, on average for a set of SPEC CPU2006 workloads when compared to a baseline without any page migration and a system with on-the-fly page migration using OS-based address reconciliation, respectively. Furthermore, we present an analytical model for classifying applications as page migration friendly (applications that show performance gains from page migration) or unfriendly based on memory access behavior.

Design, Modeling, and Analysis of Memory Allocation Policies for Rack-Scale Memory Disaggregation

“…Accessing a cache block from remote memory incurs lesser latency (150-200 ns) compared to page-based access [1] but brings up a small amount of data to a local cache. On the other hand, page-based access (or page-swap) not only introduces extra latency overhead (1.5-2 µs) but also stalls the CPU while performing address reconciliation [22] at the compute node during page-swap. Address reconciliation includes updating the page-table address translations, TLB-shootdown, and cache tag update on a page swap.…”