VM-based shared memory on low-latency, remote-memory-access networks

Abstract

Recent technological advances have produced network interfaces that provide users with very low-latency access to the memory of remote machines. We examine the impact of such networks on the implementation and performance of software DSM. Specifically, we compare two DSM systems-Cashmere and TreadMarks--on a 32-processor DEC Alpha cluster connected by a Memory Channel network.Both Cashmere and TreadMarks use virtual memory to maintain coherence on pages, and both use lazy, multi-writer release consistency. The systems differ dramatically, however, in the mechanisms used to track sharing information and to collect and merge concurrent updates to a page, with the result that Cashmere communicates much more frequently, and at a much finer grain* Our principal conclusion is that low-latency networks make DSM based on fine-grain communication competitive with more coarse-grainapproaches,butthatfurtherhardwareimprovements will be needed before such systems can provide consistently superior performance. In our experiments, Cashmere scales slightly better than TreadMarks for applications with false sharing. At the same time, it is severely constrainedby limitations of the current Memory Channel hardware. In general, performance is better for TreadMarks.

An OS-based alternative to full hardware coherence on tiled CMPs

“…While this mechanism may seem very similar to previous software cache coherence mechanisms (e.g., [18]), it differs from these in one crucial way. Namely, it does not allow multiple writers, reverts to a single up-to-date copy of every page upon a write, and enforces remote cache accesses in such cases.…”

“…However, that system also allowed indiscriminate private caching of data and no details are given on how coherence would be maintained, and it leaves the decision of caching versus remote accesses to the programmer/compiler. Finally, our work is also related to previous work on software DSM systems, such as [5,15,18,23]. Similarly to our proposal, those also tried to avoid the costs of hardware coherence by using the OS page mechanism to enforce coherence, but unlike ours, the majority of those systems supported full-blown coherence in software with full replication and multiple readers and writers.…”

“…An alternative to enforce coherence in a distributed memory system is to use the OS' virtual memory (VM) system to handle the copies of virtual pages, as was done on software DSM systems (e.g., [5,15,18,23]). In this scheme, all caches are private and it is the responsibility of software to maintain coherence.…”

“…While [23] enforced a single-writer policy, it allowed ownership to move across nodes instead of enforcing remote accesses, which can lead to significant traffic. In addition, while [18] supported remote writes, it did not support remote reads, which had to be implemented by a tortuous mechanism by which the remote node performs remote writes on request. Those works also differ from ours in that they were tailored to multi-computer systems, where no hardware-supported single address exists.…”

Removing the overhead from software-based shared memory

“…Several other papers have suggested hardware support for fine-grain remote write operations in the network interface [23], [22]. One of the recent implementations is the automatic update release consistency (AURC) home-based protocol [16].…”