Fast, Small, and Simple Document Listing on Repetitive Text Collections

Abstract

Document listing on string collections is the task of finding all documents where a pattern appears. It is regarded as the most fundamental document retrieval problem, and is useful in various applications. Many of the fastest-growing string collections are composed of very similar documents, such as versioned code and document collections, genome repositories, etc. Plain pattern-matching indexes designed for repetitive text collections achieve orders-of-magnitude reductions in space. Instead, there are not many analogous indexes for document retrieval. In this paper we present a simple document listing index for repetitive string collections of total length n that lists the ndoc distinct documents where a pattern of length m appears in time O(m + ndoc · lg n). We exploit the repetitiveness of the document array (i.e., the suffix array coarsened to document identifiers) to grammar-compress it while precomputing the answers to nonterminals, and store them in grammar-compressed form as well. Our experimental results show that our index sharply outperforms existing alternatives in the space/time tradeoff map.Muthunkishnan [20] designed the first linear-space and optimal-time index for general string collections. Given a collection of total length n, he builds an index of O(n) words that lists the ndoc documents where a pattern of length m appears in time O(m + ndoc). While linear space is deemed as sufficiently small in classic scenarios, the solution is impractical for very large text collections unless one resorts to disk, which is orders of magnitude slower. Sadakane [26] showed how to reduce the space of Muthukrishnan's index to that of the statistically-compressed text plus O(n) bits, while raising the time complexity to only O(m + ndoc · lg n) if the appropriate underlying pattern-matching index is used [2].The sharp growth of text collections is a concern in many recent applications, outperforming Moore's Law in some cases [27]. Fortunately, many of the fastest-growing text collections are highly repetitive: each document can be obtained from a few large blocks of other documents. These collections arise in different areas, such as repositories of genomes of the same species (which differ from each other by a small percentage only) like the 100K-genome project 1 , software repositories that store all the versions of the code arranged in a tree or acyclic graph like GitHub 2 , versioned document repositories where each document has a timeline of versions like Wikipedia 3 , etc. On such text collections, statistical compression is ineffective [14] and even O(n) bits of extra space can be unaffordable.Repetitiveness is the key to tackle the fast growth of these collections: their amount of new material grows much slower than their size. For example, version control systems compress those collections by storing the list of edits with respect to some reference document that is stored in plain form, and reconstruct it by applying the edits to the reference version. Much more challenging, however, is to index those ...

Tailoring r-index for metagenomics

“…Following the ideas for the document listing problem proposed in [9], we grammar compress DA producing a binary and balanced grammar of ν non-terminals, that can be stored in O(r log(n/r)) bits [12]. Let T be the parse tree of the document array DA [1..n], given a non terminal node nt ∈ T let DA[s nt ..e nt ] be its expansion.…”

“…• GCDA-PDL: Grammar-Compressed Document Array with Precomputed Document Lists. Solution described in Section 4.1, using balanced Re-Pair 2 for DA and sampling the sparse tree as in [9].…”

“…All solutions use an r-index [12] as text index for the concatenation of all documents. The first solution is an extension to frequencies of the solution proposed in [9] in which a grammar-compressed document array is used, and for each non terminal node, precomputed answers are stored. The second and the third solution are based on the term frequency approach presented in [34] which uses an additional index for all documents.…”

“…The key idea is to find the leftmost and rightmost occurrence of the pattern P in the index of each document, by searching the pattern in the index of the concatenation of all documents. To do this, the second solution uses the grammar-compressed document array of [9] enhanced with bitvectors at non terminal nodes marking which descendant contains the leftmost and rightmost occurrence of the pattern in each document. The third solution relies on a modified version of the inter-leaved longest common prefix array [13].…”

The General Higher-Order Neural Network Model and Its Application to the Archive Retrieval in Modern Guangdong Customs Archives

“…Information retrieval is a process and a technique that allows information users to find relevant information they need through a dataset, where information is organized in a certain way [1,2,3]. A document retrieval system contains mainly three parts: index generation, query processing, and document retrieval.…”