Data compression in full-text retrieval systems

Abstract

When data compression is applied to full-text retrieval systems, intricate relationships emerge between the amount of compression, access speed, and computing resources required. We propose compression methods, and explore corresponding tradeoffs, for all components of static full-text systems such as text databases on CD-ROM. These components include lexical indexes, inverted files, bitmaps, signature files, and the main text itself. Results are reported on the application of the methods to several substantial full-text databases, and show that a large, unindexed text can be stored, along with indexes that facilitate fast searching, in less than half its original size-at some appreciable cost in primary memory requirements.

Self-indexing inverted files for fast text retrieval

“…In the absence of compression four bytes and two bytes respectively might be allocated for the d and f d,t values, that is, six bytes for each d, f d,t pair. Using compression the space required can be reduced to about one byte per pair [1]. On the 2 Gb TREC collection, described below, these methods compress the inverted file from 1100 Mb to 184 Mb, an irresistible saving.…”

“…Without compression, an inverted file can easily be as large or larger than the text it indexes. Compression results in a net space reduction of as much as 80% of the inverted file size [1], but even with fast decompression-decoding at approximately 400,000 numbers per second on a Sun Sparc 10-it involves a substantial overhead on processing time.…”

Compression of nucleotide databases for fast searching

“…However, because databases are divided into records that must be independently decompressible , adaptive techniques are generally not effective. Similarly, arithmetic coding is in general the preferred coding technique; but it is slow for database applications (Bell et al, 1993).…”

“…We have used the Elias gamma codes to encode each count w and Golomb codes to represent each sequence of offsets. These techniques are a variation on techniques used for inverted file compression, which has been successfully applied to large text databases (Bell et al, 1993) and to genomic databases (Williams and Zobel, 1996a;Williams and Zobel, 1996b).…”

Memory efficient ranking

“…On the two gigabyte TREC collection these techniques compress the inverted file from 1000 megabytes to 135 megabytes, a dramatic saving. For this reason, if the information retrieval system is to be available on CD-ROM, and if we wish to maximise the amount of information stored on each disk, we should employ compression of both the index and also the stored text [2,3,9]. This is the environment that we consider here.…”