Extending On-chip Interconnects for rack-level remote resource access

Design, Modeling, and Analysis of Memory Allocation Policies for Rack-Scale Memory Disaggregation

“…A programming model similar to RDMA-style queuepair programming was implemented to access off-chip remote memory through coherent on-chip interconnects. Venice [5] and DEOI [6] also explored similar on-chip memory-semantic interconnects for remote memory access that can support both fine-grained and coarse-grained memory access over separate channels. Recently, a consortium of hardware industry leaders released protocol standards and specifications for a similar on-chip memory coherent interconnect Gen-Z that also includes a switch, and a pooled memory subsystem [18].…”

“…Due to the unavailability of a full-scale hardware disaggregated system, we follow an incremental approach to validate different components of our simulator. The integrated on-chip interconnects, such as GenZ, are either unavailable or only have been tested with small-scale prototypes using FPGAs [12,31,6]. In this section, we validate the core and cache subsystems of DRackSim against Gem5 system emulation mode (SE) and show the impact of network interconnect separately through our experiments in the next section.…”

“…Cache-based access requires a memory binding fabric support for faster access [5][6][7] to avoid the passes through network protocol stack and slow I/O buses while filling cache lines. On the other hand, accessing remote memory in larger chunks, such as a memory page, is best supported by remote direct memory access (RDMA) [8,9] like protocols.…”

“…The local and remote memory address space is organized in a flat-address structure at compute nodes, while the remote memory is accessed through a memory-centric interconnect integrated on-chip. Such interconnects have been proposed earlier for low latency and high bandwidth access to remote memory [6,21,31]. GenZ [15,12] is the latest interconnect architecture that allows such memory-centric access to fabric-attached memory (FAM).…”

“…Without commercially available disaggregated memory systems, a simulation framework is required to translate research ideas into working models by getting enough insight into the system's performance and trade-offs. The focus of disaggregated memory is on data centers and high-performance computing (HPC) facilities where disaggregation inside a rack is considered practically feasible [6,16,22,24]. A rack-scale disaggregated simulation primarily requires modeling compute nodes, memory pools, and other components, such as a memory manager and an interconnect.…”