Using network interface support to avoid asynchronous protocol processing in shared virtual memory systems

Abstract

The performance of page-based software shared virtual memory (SVM) is still far from that achieved on hardware-coherent distributed shared memory (DSM) systems. The interrupt cost for asynchronous protocol processing has been found to be a key source of performance loss and complexity.This paper shows that by providing simple and general support for asynchronous message handling in a commodity network interface (NI), and by altering SVM protocols appropriately, protocol activity can be decoupled from asynchronous message handling and the need for interrupts or polling can be eliminated. The NI mechanisms needed are generic, not SVM-dependent. They also require neither visibility into the node memory system nor code instrumentation to identify memory operations. We prototype the mechanisms and such a synchronous home-based LRC protocol, called GeNIMA (GEneral-purpose Network Interface support in a shared Memory Abstraction), on a cluster of SMPs with a programmable NI, though the mechanisms are simple and do not require programmability.We find that the performance improvements are substantial, bringing performance on a small-scale SMP cluster much closer to that of hardware-coherent shared memory for many applications, and we show the value of each of the mechanisms in different applications. Application performance improves by about 37% on average for reasonably well performing applications, even on our relatively slow programmable NI, and more for others. We discuss the key remaining bottlenecks at the protocol level and use a firmware performance monitor in the NI to understand the interactions with and the implications for the communication layer.

Removing the overhead from software-based shared memory

“…It should be possible to manually modify those applications to solve this problem. 5 We began all measurements at the start of the parallel phase to exclude DSZOOM-WF's run-time system initialization time.…”

“…Efficiency overhead for sequential execution, presented in parentheses in the last column, is between 1.03 and 1.59 (which averages 1.22) for all of the studied applications. Thus, in- 5 Artiaga has experienced similar problems with the original fork-exec versions of Volrend and Cholesky [1].…”

“…Floating-point store snippets are the major slowdown factor for FFT, LU-c, and LU-nc. LU is one of the most store-intensive SPLASH-2 applications [43] and will typically perform much better on software-based DSM systems with weaker memory models (for example on GeNIMA [5] with home-based LRC protocol). Table 2: Instrumentation overheads for parallel execution on a single-node DSZOOM-WF system.…”

“…Many different SW-DSM implementations have been proposed over the years: Blizzard-S [38], Brazos [40], Cashmere- 2L [41], [10], CRL [19], GeNIMA [5], Ivy [26], [27], MGS [44], Munin [8], Shasta [35], [34], [32], [33], [10], Sirocco-S [36], SoftFLASH [11], and TreadMarks [21]. Most of them suffer from synchronous interrupt protocol processing.…”

“…While other work has proposed elaborate schemes for cutting down on the overhead associated with interrupting and/or polling caused by the asynchronous communication between the agents [5], [29], our implementation has completely eliminated the protocol-agent interactions. In DSZOOM the entire coherence protocol is implemented in the protocol handler running in the requesting processor.…”