Pin-down cache: a virtual memory management technique for zero-copy communication

Abstract

The overhead of copying data through the central processor by a message passing protocol limits data transfer bandwidth. If the network intelface directly transfers the user's memory to the network by issuing DMA, such data copies may be eliminated. Since the DMA facility accesses the physical memory address space, user virtual memory must be pinned down to a physical memory location before the message is sent or received. If each message transfer involves pin-down and release kernel primitives, message transfer bandwidth will decrease since those primitives are quite expensive. We propose a zero copy message transfer with a pin-down cache technique which reuses the pinneddown area to decrease the number of calls to pin-down and release primitives. The proposed facility has been implemented in the PM low-level communication library on our RWC PC Cluster II, consisting of 64 Pentium Pro 200 MHz CPUs connected by a Myricom Myrinet network, and running NetBSD. The PM achieves 108.8 MByteshec for a 100 % pin-down cache hit ratio and 78.7MByteskec for all pindown cache miss. The MPI library has been implemented on top of PM. According to the NAS Parallel benchmarks result, an application is still better performance in case that cache miss ratio is very high.

Scalable RDMA performance in PGAS languages

“…Long messages are transferred with an MPIlike rendezvous protocol with memory registration/deregistration embedded into the phases of the protocol. As an optimization a cache of registered memory regions was implemented [26] with lazy memory deregistration.…”

PM2: High Performance Communication Middleware for Heterogeneous Network Environments

“…Following the previous PM implementation [7], the pin-down cache technique is employed in PM2. The pin-down cache technique reuses the pinned-down area to decrease the number of calls to pin-down and release primitives [7].…”

“…The pin-down cache technique reuses the pinned-down area to decrease the number of calls to pin-down and release primitives [7].…”

Decoupling memory pinning from the application with overlapped on-demand pinning and MMU notifiers

“…This memory pinning is known to be very expensive [9]. It has been the subject of many research projects which led to the idea of caching it [15]. It reduces the observed overhead if the applications reuse the same buffers several times, but the overhead often remains high.…”