Active Memory Clusters: Efficient Multiprocessing on Commodity Clusters

Abstract

Abstract. We show how novel active memory system research and system networking trends can be combined to realize hardware distributed shared memory on clusters of industry-standard workstations. Our active memory controller extends the cache coherence protocol to support transparent use of address re-mapping techniques that dramatically improve single-node performance, and also contains the necessary functionality for building a hardware DSM machine. Simultaneously, commodity network technology is becoming more tightly-integrated with the memory controller. We call our design of active memory commodity nodes interconnected by a next-generation network active memory clusters. We present a detailed design of the AMC architecture, focusing on the active memory controller and the network characteristics necessary to support AMC. We show simulation results for a range of parallel applications, showing that AMC performance is comparable to that of custom hardware DSM systems and far exceeds that of the fastest software DSM solutions.

Architectural support for uniprocessor and multiprocessor active memory systems

“…Instead, our system handles the messages entirely in the memory controller, similar to a hardware DSM machine. The "normal" (not using active memory support) parallel performance of this Active Memory Cluster (AMC) configuration is detailed in [11].…”

“…Note that, for normal memory operations, our active memory protocol has only the additional overhead of checking whether the AM bit is set or not. This does not slow down conventional applications since this check is not on the critical path [11], [15].…”

Active memory techniques for ccNUMA multiprocessors

“…To achieve high performance, we run the protocol code on a customized dualissue embedded protocol processor augmented with a special data path unit satisfying the needs of active memory operations [9]. To support a multi-node cluster architecture, introduced in [6] as Active Memory Clusters (AMC), our memory controller is equipped with a network interface in addition to a processor interface. In forthcoming network architectures such as InfiniBand [7], or 3GIO [17], the network interface will be moved closer to the main CPU, attached directly to or integrated with the memory controller.…”

“…Some initial results of running non-active memory SPLASH-2 applications [18] are presented in [6], which shows that our Active Memory Cluster system has comparable performance to hardware DSM systems for non-active memory applications (the small differences are due to the network speed and do not affect single-node systems). The main contribution of this paper over our previous work is that it investigates the scalability of the active memory techniques in multi-node DSM systems and the corresponding DSM protocol issues, while [9] focuses on the role of the cache coherence protocol in single-node active memory systems and [6] on the performance of our memory controller for non-active memory applications.…”

“…In this paper, we expand our previous work to multi-node hardware distributed shared memory (DSM) systems using the same active memory controller with an integrated commodity network interface, while changing only the protocol software that is run on the memory controller. We call the resulting multi-node systems Active Memory Clusters (AMC), whose architecture is introduced in [6].…”