Turning Centralized Coherence and Distributed Critical-Section Execution on their Head

Abstract

A coherent global address space in a distributed system enables shared memory programming in a much larger scale than a single multicore or a single SMP. Without dedicated hardware support at this scale, the solution is a software distributed shared memory (DSM) system. However, traditional approaches to coherence (centralized via "active" home-node directories) and critical-section execution (distributed across nodes and cores) are inherently unfit for such a scenario. Instead, it is crucial to make decisions locally and avoid the long latencies imposed by both network and software message handlers. Likewise, synchronization is fast if it rarely involves communication with distant nodes (or even other sockets). To minimize the amount of long-latency communication required in both coherence and critical section execution, we propose a DSM system with a novel coherence protocol, and a novel hierarchical queue delegation locking approach. More specifically, we propose an approach, suitable for data-race-free (DRF) programs, based on selfinvalidation, self-downgrade, and passive data classification directories that require no message handlers, thereby incurring no extra latency. For fast synchronization we extend Queue Delegation Locking to execute critical sections in large batches on a single core before passing execution along to other cores, sockets, or nodes, in that hierarchical order. The result is a software DSM system called Argo which localizes as many decisions as possible and allows high parallel performance with little overhead on synchronization when compared to prior DSM implementations.

Efficient distributed memory management with RDMA and caching

“…For comparison, we use two baselines, Argo [24] and GAM-TSO. GAM-TSO is a variant of GAM which appends a MFence operation to each write to disable write reorder and hence enforce TSO consistency model.…”

“…GAM-TSO is a variant of GAM which appends a MFence operation to each write to disable write reorder and hence enforce TSO consistency model. We compare GAM against Argo [24] and GAM-TSO in terms of the throughput achieved under different workloads. Since GAM-TSO only affects the processing of write requests, its lock throughput is the same as GAM, and hence omitted.…”

“…We compare our GAM-based DHT implementation with the DHT implementations built on top of other RDMA-based shared memory frameworks, including Grappa [35] and Argo [24]. We use these DHTs to run the YCSB benchmark [12] over 8 nodes.…”

“…As shown in Table 1, the throughput of current InfiniBand RDMA technology (e.g., 200 Gb/s Mellanox ConnectX @ -6 EN Adapter [31]) is almost approaching that of local memory access, and can be even better than NUMA (Non-Uniform Memory Access) inter-node communication channels (e.g., QPI [34]). Thus, several RDMA-based systems [14,24,30,35] have been proposed to leverage RDMA to enable a unified memory abstraction from physically distributed memory nodes. However, they still require users to manually call synchronization primitives for cache consistency, and hence suffer from the same problems as the traditional DSM systems [4,7,26,40].…”

“…• We build two applications: a transaction engine and a distributed hash table (DHT), by using the APIs provided by GAM, to demonstrate how GAM can be used for building high-level applications. • We conduct an extensive micro benchmark to profile the performance of GAM from various perspectives, and a macro benchmark to show the superior performance of GAM over L-Store [27], FaRM [14], Tell [30], Grappa [35] and Argo [24] in terms of distributed transaction processing and DHT.…”