Asynchronous memory access chaining

Abstract

In-memory databases rely on pointer-intensive data structures to quickly locate data in memory. A single lookup operation in such data structures often exhibits long-latency memory stalls due to dependent pointer dereferences. Hiding the memory latency by launching additional memory accesses for other lookups is an effective way of improving performance of pointer-chasing codes (e.g., hash table probes, tree traversals). The ability to exploit such inter-lookup parallelism is beyond the reach of modern out-of-order cores due to the limited size of their instruction window. Instead, recent work has proposed software prefetching techniques that exploit inter-lookup parallelism by arranging a set of independent lookups into a group or a pipeline, and navigate their respective pointer chains in a synchronized fashion. While these techniques work well for highly regular access patterns, they break down in the face of irregularity across lookups. Such irregularity includes variable-length pointer chains, early exit, and read/write dependencies.This work introduces Asynchronous Memory Access Chaining (AMAC), a new approach for exploiting interlookup parallelism to hide the memory access latency. AMAC achieves high dynamism in dealing with irregularity across lookups by maintaining the state of each lookup separately from that of other lookups. This feature enables AMAC to initiate a new lookup as soon as any of the in-flight lookups complete. In contrast, the static arrangement of lookups into a group or pipeline in existing techniques precludes such adaptivity. Our results show that AMAC matches or outperforms state-of-the-art prefetching techniques on regular access patterns, while delivering up to 2.3x higher performance under irregular data structure lookups. AMAC fully utilizes the available microarchitectural resources, generating the maximum number of memory accesses allowed by hardware in both single-and multi-threaded execution modes.

Interleaving with coroutines

“…Having demonstrated interleaving with coroutines on the distinct codepaths of binary search and CSB + -tree traversal, we believe our technique applies to the lookup methods of any pointer-based index. A hash-table with bucket lists is such an index, so the probe phases of hash joins that use it are straighforward candidates for our technique; moreover, since Kocberber et al [15] demonstrate AMAC also on the build phase, and our technique is equivalent to AMAC, interleaving with coroutines applies also to important hash-join operators. In fact, our technique can be employed in any data-or task-parallel operation with a memory stall problem, like sort operators, or operations related to the state management of the lock and the transaction manager.…”

“…Coroutines can yield control in the middle of their execution and be later resumed. This ability makes them candidates for implementing ISI, as already remarked by Kocberber et al [15]. An efficient implementation needs a) a suspension/resumption mechanism that consumes a few tens of cycles at most, and b) a space footprint that does not thrash the cache.…”

“…Kocberber et al [15] considered this coupling as a limitation that complicates cases where each instruction stream 2 We have not yet investigated how to form a pipeline with variable size, so we do not provide a SPP implementation. follows a different control flow.…”

“…• The runInterleaved scheduler initializes a group of coroutines, specifying interleaved=true, and maintains a buffer of coroutine handles (lines [15][16][17]. Since lookup execution now suspends, the while loop over the buffer resumes unfinished lookups (line 23), or retrieves the results from the finished lookups (lines [25][26] and starts new ones (lines 27-29).…”

“…Prior works propose two forms of such instruction stream interleaving (ISI): static, like group prefetching (GP) [6] and software pipelined prefetching (SPP) [6], and dynamic, like the state-of-the-art asynchronous memory access chaining (AMAC) [15]. Static interleaving has negligible overhead for instruction streams with identical control flow, whereas dynamic interleaving efficiently supports a wider range of use cases, allowing instruction streams to diverge, e.g., with early returns.…”