Complete inverted files for efficient text retrieval and analysis

Abstract

Abstract. Given a finite set of texts S = (wi , *.., wk) over some fixed finite alphabet 2, a complete inverted tile for S is an abstract data type that provides the functionsfind( which returns the longest prefix of w that occurs (as a subword of a word) in S, freq(w), which returns the number of times w occurs in S, and locations(w), which returns the set of positions where w occurs in S. A data structure. that implements a complete inverted file for S that occupies linear space and can be built in linear time, using the uniform-cost RAM model, is given. Using this data structure, the time for each of the above query functions is optimal. To accomplish this, techniques from the theory of finite automata and the work on suffix trees are used to build a deterministic finite automaton that recognizes the set of all subwords of the set S. This automaton is then annotated with additional information and compacted to facilitate the desired query functions. The result is a data structure that is smaller and more flexible than the s&ix tree.

Efficient Computation of Substring Equivalence Classes with Suffix Arrays

“…In this subsection, we recall the equivalence relations introduced by Blumer et al [10,1], and then state their properties. Throughout this paper, we consider the equivalence classes of the input string w that ends with a distinct symbol $ that does not appear anywhere else in w. For any string x ∈ Substr(w), let,…”

“…However, a given text contains too many substrings to browse or analyze. A reasonable approach is to partition the set of substrings into equivalence classes under the equivalence relation of [1] so that an expert can examine the classes one by one [3]. This equivalence relation groups together substrings that correspond to essentially identical occurrences in the text.…”

“…Thus, we consider these succinct expressions of the equivalence classes, which require only O(n) space. The succinct expressions can easily be computed using the CDAWG data structure proposed by [1], which is an acyclic graph structure whose nodes correspond to the equivalence classes. Although CDAWGs can be constructed in O(n) time and space [4], we present a more efficient algorithm based on suffix arrays.…”

“…This paper considers enumeration of substring equivalence classes introduced by Blumer et al [1]. They used the equivalence classes to define an index structure called compact directed acyclic word graphs (CDAWGs).…”

Contracted Suffix Trees: A Simple and Dynamic Text Indexing Data Structure

“…Previous data structures for this problem include the suffix tree [1], the compact directed acyclic word graph (compact DAWG) [2], and the suffix array [3]. The first two approaches take O(n) time to build the data structure, and O(m + k) time to find the k positions where the pattern string occurs.…”