Virtual Cache Line: A New Technique to Improve Cache Exploitation for Recursive Data Structures

Abstract

Abstract. Recursive data structures (lists, trees, graphs, etc.) are used throughout scientific and commercial software. The common approach is to allocate storage to the individual nodes of such structures dynamically, maintaining the logical connection between them via pointers. Once such a data structure goes through a sequence of updates (inserts and deletes), it may get scattered all over memory yielding poor spatial locality, which in turn introduces many cache misses. In this paper we present the new concept of Virtual Cache Lines (VCLs). Basically, the mechanism keeps groups of consecutive nodes in close proximity, forming virtual cache lines, while allowing the groups to be stored arbitrarily far away from each other. Virtual cache lines increase the spatial locality of the given data structure resulting in better locality of references. Furthermore, since the spatial locality is improved, software prefetching becomes much more attractive. Indeed, we also present a software prefetching algorithm that can be used when dealing with VCLs resulting in even higher data cache performance. Our results show that the average performance of linked list operations, like scan, insert, and delete can be improved by more than 200% even in architectures that do not support prefetching, like the Intel Pentium. Moreover, when using prefetching one can gain additional 100% improvement. We believe that given a program that manipulates certain recursive data structures, compilers will be able to generate VCL-based code. Also, until this vision becomes true, VCLs can be used to build more efficient user libraries, operating-systems and applications programs.

A Compile/Run-time Environment for the Automatic Transformation of Linked List Data Structures

“…This is a sparse array structure which allows for efficient insertion and deletion of elements while preserving locality. A similar approach is taken by Rubin et al [27], who describe a data structure that maintains groups of adjacent linked list nodes in such a way that they are loaded into a cache line simultaneously. This concept is called virtual cache lines.…”

An analysis on the performance of hash table-based dictionary implementations with different data usage models

“…Chilimbi (1999) suggested packing the homogeneous nodes of a BST into contiguous blocks of memory. Rubin et al (1999) proposed a similar idea for linked lists. These techniques do not eliminate pointers, but improve spatial access locality by grouping nodes that are likely to be accessed contemporaneously (Askitis, 2007).…”

An efficient profile-analysis framework for data-layout optimizations

“…However, when the code is too complex to be transformed-a situation common in programs that manipulate pointer-based data structures-one must resort to transforming the layout of data structures, improving spatial locality. Many data-layout optimizations have been proposed [3,4,7,8,9,11,15,16,21,23,24], with their specific goals ranging from reordering structure fields [7] to object inlining [11].…”

“…The insufficiency of static analysis information has been recognized by existing data-layout optimizations, which are all either profile-guided or exploit programmer-supplied application knowledge [3,7,8,9,11,15,16,21,23,24]. Although many of these techniques manage to sidestep the problem of analyzing the program's memory behavior by observing it at run-time, they are still fundamentally limited by the hard problem of synthesizing a good layout for the observed behavior [20].…”