Building a complete inverted file for a set of text files in linear time

Abstract

Given a finite set of texts S = [~, ..... ~vt] over some fixed finite alphabe t E, a complete inverted file for S is an abstract data type that provides the functions find('w), which returns the longest prefix of ~# which occurs in S; freq(zo), which returns the number of times ~ occurs in S; and loc=t~ons(~v) which returns the set of positions at which ~ occurs. We give a data structure to implement a complete inverted file for S which occupies linear space and can be built in linear time, using the uniform cost RAM model. Using this data structure, the time for each of the above query functions is optimal. To accomplish this, we use techniques from the theory of finite automata to build a deterministic finite automaton which recognizes the set of all subwords of the set S. This automaton is then annotated with additional information and compacted to facilitate the desired query functions.

Textual and visual access to a computer by people who know nothing about it

“…We used a software package originally designed for the Molecular, Cellular, Developmental Biology Department at the University of Colorado for DNA analysis (Blumer, Blumer, Ehrenfeucht, Haussler, & McConnell, 1984, Baggett, Ehrenfeucht, & Perry, 1986. It allows one to find and display all repeated segments, together with their contexts, on text files up to 100,000 characters.…”