A compression scheme for large databases

Cannane, Adam; Williams, Hugh E.

doi:10.1109/adc.2000.819807

Cited by 12 publications

(8 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach is not viable for large document collections as a suffix array and other auxiliary data structures of the complete collection are required for encoding. Another set of related techniques are grammar compressors, such as ray [10], xray [11] and re-pair [21]. Grammar compressors can achieve powerful compression but have enormous construction requirements, limiting their application to smaller collections.…”

Section: Adaptive Methodsmentioning

confidence: 99%

Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections

2011

View full text Add to dashboard Cite

Compression techniques that support fast random access are a core component of any information system. Current stateof-the-art methods group documents into fixed-sized blocks and compress each block with a general-purpose adaptive algorithm such as gzip. Random access to a specific document then requires decompression of a block. The choice of block size is critical: it trades between compression effectiveness and document retrieval times. In this paper we present a scalable compression method for large document collections that allows fast random access. We build a representative sample of the collection and use it as a dictionary in a LZ77-like encoding of the rest of the collection, relative to the dictionary. We demonstrate on large collections, that using a dictionary as small as 0.1% of the collection size, our algorithm is dramatically faster than previous methods, and in general gives much better compression.

show abstract

Section: Adaptive Methodsmentioning

confidence: 99%

Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections

2011

View full text Add to dashboard Cite

show abstract

“…For example, the researches by Cormack (1985) and Moffat and Zobel (1997) are based on Huffman Coding. Cannane and Williams (2000) and Cockshott et al (1998) proposed database compression methods based on Dictionary Coding, while Ng and Ravishankar (1997) proposed a research based on Differencing Cording. Linoff and Stanfill (1993) proposed a database compression method that combines Differencing Coding, Run Length Coding, and Arithmetic Coding.…”

Section: Data Compression and Database Compressionmentioning

confidence: 99%

A data mining approach to database compression

et al. 2006

View full text Add to dashboard Cite

Data mining can dig out valuable information from databases to assist a business in approaching knowledge discovery and improving business intelligence. Database stores large structured data. The amount of data increases due to the advanced database technology and extensive use of information systems. Despite the price drop of storage devices, it is still important to develop efficient techniques for database compression. This paper develops a database compression method by eliminating redundant data, which often exist in transaction database. The proposed approach uses a data mining structure to extract association rules from a database. Redundant data will then be replaced by means of compression rules. A heuristic method is designed to resolve the conflicts of the compression rules. To prove its efficiency and effectiveness, the proposed approach is compared with two other database compression methods

show abstract

“…We have previously described a preliminary implementation of XRAY elsewhere [Cannane and Williams 2000]. Our scheme is a three-phase approach based on the modeling, probability estimation, and coding paradigm discussed in Section 2.1.…”

Section: Xray-extended Raymentioning

confidence: 99%

A general-purpose compression scheme for large collections

Cannane

Williams

2002

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Compression of large collections can lead to improvements in retrieval times by offsetting the CPU decompression costs with the cost of seeking and retrieving data from disk. We propose a semistatic phrase-based approach called XRAY that builds a model offline using sample training data extracted from a collection, and then compresses the entire collection online in a single pass. The particular benefits of XRAY are that it can be used in applications where individual records or documents must be decompressed, and that decompression is fast. The XRAY scheme also allows new data to be added to a collection without modifying the semistatic model. Moreover, XRAY can be used to compress general-purpose data such as genomic, scientific, image, and geographic collections without prior knowledge of the structure of the data. We show that XRAY is effective on both text and general-purpose collections. In general, XRAY is more effective than the popular GZIP and COMPRESS schemes, while being marginally less effective than BZIP2. We also show that XRAY is efficient: of the popular schemes we tested, it is typically only slower than GZIP in decompression. Moreover, the query evaluation costs of retrieval of documents from a large collection with our search engine is improved by more than 30% when XRAY is incorporated compared to an uncompressed approach. We use simple techniques for obtaining the training data from the collection to be compressed and show that with just over 4% of data the entire collection can be effectively compressed. We also propose four schemes for phrase-match selection during the single pass compression of the collection. We conclude that with these novel approaches XRAY is a fast and effective scheme for compression and decompression of large general-purpose collections.

show abstract

A compression scheme for large databases

Cited by 12 publications

References 12 publications

Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections

Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections

A data mining approach to database compression

A general-purpose compression scheme for large collections

Contact Info

Product

Resources

About