Optimizing positional index structures for versioned document collections

Abstract

Versioned document collections are collections that contain multiple versions of each document. Important examples are Web archives, Wikipedia and other wikis, or source code and documents maintained in revision control systems. Versioned document collections can become very large, due to the need to retain past versions, but there is also a lot of redundancy between versions that can be exploited. Thus, versioned document collections are usually stored using special differential (delta) compression techniques, and a number of researchers have recently studied how to exploit this redundancy to obtain more succinct full-text index structures.In this paper, we study index organization and compression techniques for such versioned full-text index structures. In particular, we focus on the case of positional index structures, while most previous work has focused on the non-positional case. Building on earlier work in [32], we propose a framework for indexing and querying in versioned document collections that integrates non-positional and positional indexes to enable fast top-k query processing. Within this framework, we define and study the problem of minimizing positional index size through optimal substring partitioning. Experiments on Wikipedia and web archive data show that our techniques achieve significant reductions in index size over previous work while supporting very fast query processing.

Universal indexes for highly repetitive document collections

“…He and Suel [34] also designed a positional inverted index for the repetitive scenario. They apply a previous technique to partition documents into fragments [67] and then use their non-positional approach [36] on the fragments.…”

“…Our universal techniques, instead, also work on settings where the versions form a tree structure (as in collaborative document creation, software repositories, or phylogenetic trees), or where the versions form a continuous stream of incremental changes (as in periodic publications of technical data), or where it is unknown or unclear which documents are versions of which (as in DNA sequence databases, or near-duplicate pages in Web crawls). [34] also designed a positional inverted index for the repetitive scenario. They apply a previous technique to partition documents into fragments [67] and then use their non-positional approach [36] on the fragments.…”

“…There is a burst of recent activity in exploiting repetitiveness at the indexing structures, in order to provide fast searches in the collection within little space. Both inverted indexes for word and phrase queries over natural language texts [3,12,35,65,36,34], and other indexes for general string collections [43,16,19,20,40,28,29,26,9], have been pursued.…”

A two-stage constructive method for the unweighted minimum string cover problem

“…While the algorithms proposed by Canzar et al [3] and Ferdous et al [4] employ a set of candidate building blocks consisting of every possible substring in S, DISFY works on a reduced candidate set filtered by their frequencies in S. The frequency-based partitioning or discovering of fragments has been successfully applied previously to index and query in versioned document collections [8]. In this line, we may also highlight the similarity between the constructive procedure to find the basic building blocks in S and the process of discovering association rules, which is one of the most common data mining techniques Tsay and Chiang [14], Zhang and Zhang [15].…”

Document Retrieval on Repetitive String Collections

“…In the case of natural language, there exist various proposals to reduce the inverted index size by exploiting the text repetitiveness (Anick and Flynn, 1992;Broder et al, 2006;He et al, 2009He et al, , 2010He and Suel, 2012;Claude et al, 2016). For general string collections, the situation is much worse.…”