A compression method for inverted index and its FPGA-based decompression solution

Abstract

Reconfigurable computing based on FPGA is a promising solution to accelerate applications for web search engines. Due to the challenge of such data-intensive applications, data compression has become much more important. This paper proposes a data compression method for inverted indices, which combines the bit-level compression method -Huffman coding and a coarse-grained compression method, to achieve a balanced performance in compression ratio and decompression speed. Because an inverted index is only compressed once, the compression speed is not the major measurement for a compression method. The proposed method shows good to 21.61% compression ratio on inverted indices from a commercial search engine. This compression ratio is better than results by other existing compression methods. We also develop an efficient FPGA-based hardware decompression module, which could provide up to 996 MBps input bandwidth for the accelerator system.