Access Time Tradeoffs in Archive Compression

Abstract

Abstract. Web archives, query and proxy logs, and so on, can all be very large and highly repetitive; and are accessed only sporadically and partially, rather than continually and holistically. This type of data is ideal for compression-based archiving, provided that random-access to small fragments of the original data can be achieved without needing to decompress everything. The recent RLZ (relative Lempel Ziv) compression approach uses a semi-static model extracted from the text to be compressed, together with a greedy factorization of the whole text encoded using static integer codes. Here we demonstrate more precisely than before the scenarios in which RLZ excels. We contrast RLZ with alternatives based on block-based adaptive methods, including approaches that "prime" the encoding for each block, and measure a range of implementation options using both hard-disk (HDD) and solid-state disk (SSD) drives. For HDD, the dominant factor affecting access speed is the compression rate achieved, even when this involves larger dictionaries and larger blocks. When the data is on SSD the same effects are present, but not as markedly, and more complex trade-offs apply.

CSA++: Fast Pattern Search for Large Alphabets

“…The effect of this alphabet partitioning is better compression for the EFcoded values, which on this file are the dominant type; confirming that this option is Case Study, Text Factorization. The Relative Lempel-Ziv (RLZ) compression mechanism represents a string STR as a sequence of factors from a dictionary D, see Petri et al [22] for a description and experimental results. To greedily determine longest factors using a CSA, we take T = D r , the reverse of D, and build a compressed index.…”

Effective Construction of Relative Lempel-Ziv Dictionaries

“…So that document retrieval and access is fast, the sequence C is divided into fixed-length blocks (except for the last block), each of which is independently factored (in a left-to-right greedy manner) against the dictionary D. Factorization converts each block into a sequence of offset, length pairs, unless the next symbol is not previously seen, in which case length-zero literal "factor" is generated [5]. Petri et al [12] note that it is beneficial to impose a lower limit on the length of each factor, and following their recommendation, we employ a four-byte threshold. That is, if the greedy match from the current position i in C is of length three or fewer bytes long, then a literal that covers that factor is generated.…”

“…Each of the offset, length and literal streams is encoded separately. Previous experimentation has shown that compressing each with ZLib and using source blocks of size between 16 kiB and 64 kiB provides a good compromise between compression effectiveness and decompression speed [12]. With typical factor lengths of around 20 bytes or more, each of the three compressed integer streams generated for each block corresponds to, at most, approximately 3,000 integers.…”

“…The key advantage of RLZ is that pseudo-random access to the compressed text is possible, since each block can be independently decoded; yet compression effectiveness is relatively high, since the dictionary is drawn (in some way) from the whole of the text being stored. Recent experimentation exploring the trade-off between access efficiency and compression effectiveness showed that -even with a relatively small RAM-resident dictionary -RLZ factorization and ZLib-based encoding of the integer streams is competitive against state-of-the-art dynamic compression schemes [12].…”

“…Ineffective compression typically gives rise to a greater number of factors to be decoded: as a general pattern, the better the compression effectiveness, the faster the decoding rate. Petri et al [12] provide extensive measurements of decoding rates and random access costs on both mechanical and solid-state disk drives, and demonstrate that even on hard-disk drives, requests to access small fragments of the compressed collection can be carried out at a rate of 100 per second. We similarly omit dictionary construction and encoding times, but note that our proposal here operates at similar rates to the CARE mechanism of Tong et al [17].…”