A Universal Parallel Two-Pass MDL Context Tree Compression Algorithm

Abstract

Computing problems that handle large amounts of data necessitate the use of lossless data compression for efficient storage and transmission. We present a novel lossless universal data compression algorithm that uses parallel computational units to increase the throughput. The length-N input sequence is partitioned into B blocks. Processing each block independently of the other blocks can accelerate the computation by a factor of B, but degrades the compression quality. Instead, our approach is to first estimate the minimum description length (MDL) context tree source underlying the entire input, and then encode each of the B blocks in parallel based on the MDL source. With this two-pass approach, the compression loss incurred by using more parallel units is insignificant. Our algorithm is work-efficient, i.e., its computational complexity is O(N/B). Its redundancy is approximately B log(N/B) bits above Rissanen's lower bound on universal compression performance, with respect to any context tree source whose maximal depth is at most log(N/B). We improve the compression by using different quantizers for states of the context tree based on the number of symbols corresponding to those states. Numerical results from a prototype implementation suggest that our algorithm offers a better trade-off between compression and throughput than competing universal data compression algorithms.

Wireless Network Compression Via Memory-Enabled Overhearing Helpers

“…This was also experimentally confirmed on real data gathered from network traffic [9]. Note that a combination of memory-assisted compression and parallel compression techniques that achieve high compression rate as well as high compression speed make compression-based redundancy elimination feasible on high rate links as well [14], [15].…”

“…It turns out that the optimum value is achieved by λ = 1−κ κ n m for which the cost is obtained in (14). For κ = 1, it suffices to use the traditional two-part codes on the S-C link and ignore the memory content at M .…”

Packet-Level Network Compression: Realization and Scaling of the Network-Wide Benefits

“…where (45) holds by choosing M = N 1/g+δ . To show that the number of covered nodes is concentrated around its mean, we use Proposition 6 again with λ = αE [|∪N r δ (µ i , g)|].…”

“…We note that improving compression speed and reducing the complexity of compression while maintaining acceptable compression rate is the subject of active research in the compression community (cf. [45] where the authors improve both compression performance and compression speed by using parallel compression). In conclusion, the high-speed compression algorithms are suitable where communication throughput is high and processing power is limited, e.g., 128MB/sec for Ethernet 1 Gigabit/sec connection.…”