A Hybrid Approach to Index Maintenance in Dynamic Text Retrieval Systems

Büttcher, Stefan; Clarke, Charles L. A.

doi:10.1007/11735106_21

Cited by 13 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recent works on sequential systems are mainly focused on on-line incremental updates over disk-based, inverted indexes [7,8]. In [7], the authors propose a hybrid indexing technique. The proposed method merges small posting lists with the already existing index, while using posting list reallocation for large posting lists.…”

Section: Related Workmentioning

confidence: 99%

A Parallel Framework for In-Memory Construction of Term-Partitioned Inverted Indexes

Küçükyılmaz

Türk

Aykanat

2012

The Computer Journal

View full text Add to dashboard Cite

With the advances in cloud computing and huge RAMs provided by 64-bit architectures, it is possible to tackle large problems using memory-based solutions. Construction of term-based, partitioned, parallel inverted indexes is a communication intensive task and suitable for memory-based modeling. In this paper, we provide an efficient parallel framework for in-memory construction of term-based partitioned, inverted indexes. We show that, by utilizing an efficient bucketing scheme, we can eliminate the need for the generation of a global vocabulary. We propose and investigate assignment schemes that can reduce the communication overheads while minimizing the storage and final query processing imbalance. We also present a study on how communication among processors should be carried out with limited communication memory in order to reduce the total inversion time. We present several different communication-memory organizations and discuss their advantages and shortcomings. The conducted experiments indicate promising results.

show abstract

Section: Related Workmentioning

confidence: 99%

A Parallel Framework for In-Memory Construction of Term-Partitioned Inverted Indexes

Küçükyılmaz

Türk

Aykanat

2012

The Computer Journal

View full text Add to dashboard Cite

show abstract

“…It reduces the index building time but may increase the search time of the long terms due to posting retrievals from both the in-place and the merge-based part. Another variation of hybrid methods also exists that categorizes terms into short or long according to the total postings accumulated in the system rather than those only in memory and the merge-based part [4,5]. This variation increases the building time, but keeps the postings of each term in only one of the merge-based and in-place parts.…”

Section: Related Workmentioning

confidence: 99%

“…To the best of our knowledge this is the first time that a method simultaneously combines the above features. Previous methods distributed the infrequent terms randomly across different blocks [25] unless they managed them individually [3,27], only flushed partially the frequent terms from memory to disk [4,5], and obtained limited benefits from block-based storage management because they only considered small blocks of a few kilobytes [3,25].…”

Section: Introductionmentioning

confidence: 99%

Low-cost management of inverted files for online full-text search

Margaritis

Anastasiadis

2009

Proceedings of the 18th ACM Conference on Information and Knowledge Management

View full text Add to dashboard Cite

In dynamic environments with frequent content updates, we require online full-text search that scales to large data collections and achieves low search latency. Several recent methods that support fast incremental indexing of documents typically keep on disk multiple partial index structures that they continuously update as new documents are added. However, spreading indexing information across multiple locations on disk tends to considerably decrease the search responsiveness of the system. In the present paper, we take a fresh look at the problem of online full-text search with consideration of the architectural features of modern systems. Selective Range Flush is a greedy method that we introduce to manage the index in the system by using fixed-size blocks to organize the data on disk and dynamically keep low the cost of data transfer between memory and disk. As we experimentally demonstrate with the Proteus prototype implementation that we developed, we retrieve indexing information at latency that matches the lowest achieved by existing methods. Additionally, we reduce the total building cost by 30% in comparison to methods with similar retrieval time.

show abstract

“…Earlier this year, we have presented a family of hybrid index maintenance strategies based on this distinction between short and long lists [2]. The basic idea is rather simple: As soon as the posting list for a given term exceeds a certain length (we refer to this as the long list threshold, denoted as T ), is is declared long and moved from the merge-updated part of the on-disk index to the in-placemaintained part.…”

Section: Hybrid Index Maintenancementioning

confidence: 99%

“…Recently, we have pre- sented a family of hybrid strategies, based on a distinction between short and long posting lists [2]. Hybrid strategies maintain short posting lists following a merge-based approach, while long lists are updated in-place.…”

Section: Introductionmentioning

confidence: 99%

Hybrid index maintenance for growing text collections

Büttcher

Clarke

Lushman

2006

Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

We present a new family of hybrid index maintenance strategies to be used in on-line index construction for monotonically growing text collections. These new strategies improve upon recent results for hybrid index maintenance in dynamic text retrieval systems. Like previous techniques, our new method distinguishes between short and long posting lists: While short lists are maintained using a merge strategy, long lists are kept separate and are updated in-place. This way, costly relocations of long posting lists are avoided.We discuss the shortcomings of previous hybrid methods and give an experimental evaluation of the new technique, showing that its index maintenance performance is superior to that of the earlier methods, especially when the amount of main memory available to the indexing system is small. We also present a complexity analysis which proves that, under a Zipfian term distribution, the asymptotical number of disk accesses performed by the best hybrid maintenance strategy is linear in the size of the text collection, implying the asymptotical optimality of the proposed strategy.

show abstract

A Hybrid Approach to Index Maintenance in Dynamic Text Retrieval Systems

Cited by 13 publications

References 11 publications

A Parallel Framework for In-Memory Construction of Term-Partitioned Inverted Indexes

A Parallel Framework for In-Memory Construction of Term-Partitioned Inverted Indexes

Low-cost management of inverted files for online full-text search

Hybrid index maintenance for growing text collections

Contact Info

Product

Resources

About