Cache-oblivious range reporting with optimal queries requires superlinear space

Proceedings of the Twenty-Ninth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems

Wei

et al. 2010

The hash table, especially its external memory version, is one of the most important index structures in large databases. Assuming a truly random hash function, it is known that in a standard external hash table with block size b, searching for a particular key only takes expected average tq = 1+1/2disk accesses for any load factor α bounded away from 1. However, such near-perfect performance is achieved only when b is known and the hash table is particularly tuned for working with such a blocking. In this paper we study if it is possible to build a cache-oblivious hash table that works well with any blocking. Such a hash table will automatically perform well across all levels of the memory hierarchy and does not need any hardware-specific tuning, an important feature in autonomous databases.We first show that linear probing, a classical collision resolution strategy for hash tables, can be easily made cacheoblivious but it only achieves tq = 1 + O(α/b). Then we demonstrate that it is possible to obtain tq = 1 + 1/2 Ω(b) , thus matching the cache-aware bound, if the following two conditions hold: (a) b is a power of 2; and (b) every block starts at a memory address divisible by b. Both conditions hold on a real machine, although they are not stated in the cache-oblivious model. Interestingly, we also show that neither condition is dispensable: if either of them is removed, the best obtainable bound is tq = 1 + O(α/b), which is exactly what linear probing achieves.

Section: Related Resultsmentioning

confidence: 76%

Cache-oblivious hashing

Proceedings of the Twenty-Ninth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems

Wei

et al. 2010

“…In most cases, the cache-oblivious bounds match their cache-aware versions, and it has always be an interesting problem to see for what problems do we have a separation between the cache-oblivious model and the cache-aware model. Until today there have been only three separation results [1,3,5]; our lower bound adds to that list, furthering our understanding of cache-obliviousness.…”

Section: Related Resultsmentioning

confidence: 87%

“…Among them, the most successful approach is the cache-oblivious model [10] due to its elegance and simplicity. This model actually only features two levels of memory: a data structure is laid out in external memory and accessed in exactly the same way as in the standard two-level model, but the additional requirement is that the structure is unaware of the block size b, or equivalently, the structure is laid out in external memory in a way that works for all block sizes 1 . Thus a cache-oblivious data structure automatically works in a memory hierarchy.…”

Section: Introductionmentioning

confidence: 99%

Cache-Oblivious Hashing

Wei

et al. 2013

Algorithmica

The hash table, especially its external memory version, is one of the most important index structures in large databases. Assuming a truly random hash function, it is known that in a standard external hash table with block size b, searching for a particular key only takes expected average t q = 1 + 1/2disk accesses for any load factor α bounded away from 1. However, such near-perfect performance is achieved only when b is known and the hash table is particularly tuned for working with such a blocking. In this paper we study if it is possible to build a cache-oblivious hash table that works well with any blocking. Such a hash table will automatically perform well across all levels of the memory hierarchy and does not need any hardware-specific tuning, an important feature in autonomous databases.We first show that linear probing, a classical collision resolution strategy for hash tables, can be easily made cache-oblivious but it only achieves t q = 1 + Θ(α/b) even if a truly random hash function is used. Then we demonstrate that the block probing algorithm [20] achieves t q = 1 + 1/2 Ω(b) , thus matching the cache-aware bound, if the following two conditions hold: (a) b is a power of 2; and (b) every block starts at a memory address divisible by b. Note that the two conditions hold on a real machine, although they are not stated in the cache-oblivious model. Interestingly, we also show that neither condition is dispensable: if either of them is removed, the best obtainable bound is t q = 1 + O(α/b), which is exactly what linear probing achieves.

“…[10,24] for the I/Omodel, [22,17] for the pointer machine, [5,9,11,3,4] for the cache-oblivious and [8,21,14] for the word-RAM model. One of the main reason why the problem has seen so much attention stems from the fact that range searching with more than three sides no longer admits linear space data structures with polylogarithmic query cost and a linear term in the output size.…”

Section: Related Workmentioning

confidence: 99%

I/O-Efficient Data Structures for Colored Range and Prefix Reporting

Larsen

Proceedings of the Twenty-Third Annual ACM-SIAM Symposium on Discrete Algorithms

2012

Motivated by information retrieval applications, we consider the one-dimensional colored range reporting problem in rank space. The goal is to build a static data structure for sets C 1 , . . . , C m ⊆ {1, . . . , σ} that supports queries of the kind: Given indices a, b, report the set a≤i≤b C i .We study the problem in the I/O model, and show that there exists an optimal linear-space data structure that answers queries in O(1 + k/B) I/Os, where k denotes the output size and B the disk block size in words. In fact, we obtain the same bound for the harder problem of three-sided orthogonal range reporting. In this problem, we are to preprocess a set of n two-dimensional points in rank space, such that all points inside a query rectangle of the form [x 1 , x 2 ] × (−∞, y] can be reported. The best previous bounds for this problem is either O(n lg B n is the base B logarithm iterated h times, for any constant integer h. The previous bounds are both achieved under the indivisibility assumption, while our solution exploits the full capabilities of the underlying machine. Breaking the indivisibility assumption thus provides us with cleaner and optimal bounds.Our results also imply an optimal solution to the following colored prefix reporting problem. Given a set S of strings, each O(1) disk blocks in length, and a function c : S → 2 {1,...,σ} , support queries of the kind: Given a string p, report the set x∈S∩p * c(x), where p * denotes the set of strings with prefix p. Finally, we consider the possibility of top-k extensions of this result, and present a simple solution in a model that allows non-blocked I/O.