Hybrid index maintenance for growing text collections

Abstract

We present a new family of hybrid index maintenance strategies to be used in on-line index construction for monotonically growing text collections. These new strategies improve upon recent results for hybrid index maintenance in dynamic text retrieval systems. Like previous techniques, our new method distinguishes between short and long posting lists: While short lists are maintained using a merge strategy, long lists are kept separate and are updated in-place. This way, costly relocations of long posting lists are avoided.We discuss the shortcomings of previous hybrid methods and give an experimental evaluation of the new technique, showing that its index maintenance performance is superior to that of the earlier methods, especially when the amount of main memory available to the indexing system is small. We also present a complexity analysis which proves that, under a Zipfian term distribution, the asymptotical number of disk accesses performed by the best hybrid maintenance strategy is linear in the size of the text collection, implying the asymptotical optimality of the proposed strategy.

Workload-aware indexing for keyword search in social networks

“…An updatable keyword search system is usually implemented with a hierarchy of indexes [24,14,21,26]. New data is accumulated in a small updatable structure that also supports concurrent queries, while the main part of the hierarchy consists of a set of read-only indexes.…”

“…We use cost models to explore this large space. In information retrieval, cost models have traditionally been used to explain and compare the relative merits of different algorithms [35,34,14,16]. In this paper, however, the cost models are used in optimization algorithms to select between different access designs for a stratified workload of updates and queries.…”

“…Cost models have been used to estimate the efficiency of processing strategies in both information retrieval and databases [35,14,16,25]. There exist advanced cost models to evaluate different index construction and update strategies [14,35].…”

“…There exist advanced cost models to evaluate different index construction and update strategies [14,35]. For search queries, however, simple models are most commonly used, sometimes without verification on an actual system [16].…”

Cache-Oblivious Dictionaries and Multimaps with Negligible Failure Probability

“…More recently, Luk and Lam [23] describe an internal-memory inverted file implementation based on hash tables with chaining, but their method also does not support fast item removals. Lester et al [21,22] and Büttcher et al [10] describe externalmemory inverted file implementations that support item insertions only. Büttcher and Clarke [9] consider trade-offs for allowing for both item insertions and removals, and Guo et al [17] give a solution for performing such operations by using a B-tree variant.…”