Compact full-text indexing of versioned document collections

Abstract

We study the problem of creating highly compressed fulltext index structures for versioned document collections, that is, collections that contain multiple versions of each document. Important examples of such collections are Wikipedia or the web page archive maintained by the Internet Archive. A straightforward indexing approach would simply treat each document version as a separate document, such that index size scales linearly with the number of versions. However, several authors have recently studied approaches that exploit the significant similarities between different versions of the same document to obtain much smaller index sizes.In this paper, we propose new techniques for organizing and compressing inverted index structures for such collections. We also perform a detailed experimental comparison of new techniques and the existing techniques in the literature. Our results on an archive of the English version of Wikipedia, and on a subset of the Internet Archive collection, show significant benefits over previous approaches.

Durable top-k search in document archives

“…Indexing versioned document collections has been studied in [7,25,14,13]. Broder et al [7] propose a technique that exploits large content overlaps between documents to achieve a reduction in index size.…”

“…[25] uses content-dependent partitioning technique [21] to partition a page into smaller fragments such that more fragments are common between versions. More recent approaches by Hersovici et al [14] and He et al [13] exploit arbitrary content overlaps between documents to reduce index size. [14] attempt to find subsets of terms that are contained in consecutive versions of a document.…”

“…Each subset is stored into a virtual document and the total storage cost is optimized by minimizing the overall number and size of the virtual documents. [13] propose a two-level index compression that improves the query processing time. This approach groups similar union-documents into clusters, where a union-document contains all terms in the corresponding versions, and the terms are compressed locally for each cluster.…”

Indexes for highly repetitive document collections

“…Both inverted indexes for word and phrase queries over natural language texts [2,5,11,12], and other indexes for general string collections [16,6,14,7], have been pursued.…”

“…He et al [11,12] have presented alternative compression methods specifically targeted at highly repetitive collections. Their approach merges all versions of each document for creating the inverted lists and then keeps a secondary index that allows one to list the versions of a document that contain a given term.…”