Compression techniques for fast external sorting

Yiannis, John; Zobel, Justin

doi:10.1007/s00778-006-0005-2

Cited by 10 publications

(4 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Yiannis and Zobel studied the possibility of compressing sets of records during the run generation phase in order to reduce disk and transfer costs of external sorting by reducing the number of runs generated, and proposed a new compression technique adapted to sets of records [11].…”

Section: Related Workmentioning

confidence: 99%

Two-way replacement selection

2010

View full text Add to dashboard Cite

The performance of external sorting using merge sort is highly dependent on the length of the runs generated. One of the most commonly used run generation strategies is Replacement Selection (RS) because, on average, it generates runs that are twice the size of the memory available. However, the length of the runs generated by RS is downsized for data with certain characteristics, like inputs sorted inversely with respect to the desired output order.The goal of this paper is to propose and analyze two-way replacement selection (2WRS), which is a generalization of RS obtained by implementing two heaps instead of the single heap implemented by RS. The appropriate management of these two heaps allows generating runs larger than the memory available in a stable way, i.e. independent from the characteristics of the datasets. Depending on the changing characteristics of the input dataset, 2WRS assigns a new data record to one or the other heap, and grows or shrinks each heap, accommodating to the growing or decreasing tendency of the dataset. On average, 2WRS creates runs of at least the length generated by RS, and longer for datasets that combine increasing and decreasing data subsets. We tested both algorithms on large datasets with different characteristics and 2WRS achieves speedups at least similar to RS, and over 2.5 when RS fails to generate large runs.

show abstract

Section: Related Workmentioning

confidence: 99%

Two-way replacement selection

2010

View full text Add to dashboard Cite

show abstract

“…Sorting lexicographically large files in external memory is not excessively expensive [30,31]. With a memory buffer of M elements, we can sort almost M 2 elements in two passes.…”

Section: Sorting Rowsmentioning

confidence: 99%

Sorting improves word-aligned bitmap indexes

Lemire

Kaser

Aouiche

2010

Data & Knowledge Engineering

View full text Add to dashboard Cite

Bitmap indexes must be compressed to reduce input/output costs and minimize CPU usage. To accelerate logical operations (AND, OR, XOR) over bitmaps, we use techniques based on run-length encoding (RLE), such as Word-Aligned Hybrid (WAH) compression. These techniques are sensitive to the order of the rows: a simple lexicographical sort can divide the index size by 9 and make indexes several times faster. We investigate row-reordering heuristics. Simply permuting the columns of the table can increase the sorting efficiency by 40%. Secondary contributions include efficient algorithms to construct and aggregate bitmaps. The effect of word length is also reviewed by constructing 16-bit, 32-bit and 64-bit indexes. Using 64-bit CPUs, we find that 64-bit indexes are slightly faster than 32-bit indexes despite being nearly twice as large.

show abstract

“…However, because the size limit of main memory, a large file can not be sorted in main memory so that the external merge sort in disk-based storage systems has been studied extensively. Many previous research [1,2,3] has been proposed for the external merge sort in disk-based storage systems. Because the external merge sort in disk-based storage systems might incur a lot of data access, the external sorting design on flash-based storage systems will be different from the implementations and considerations on disk-based storage systems.…”

Section: Motivationmentioning

confidence: 99%

Data sorting in flash memory

2012

Proceedings of the 27th Annual ACM Symposium on Applied Computing

View full text Add to dashboard Cite

Since huge-capacity flash memory is now an economic solution for various portable devices and enterprise systems, a NAND flash-based storage system has replaced a hard-disk drive in many applications. Recently, the implementation of database systems on NAND flash-based storage systems has become an important research topic. In particular, the external sorting is an important operation on database systems. With the very distinctive characteristics of flash memory, the typical external sorting that adopts a clustered sorting could result in performance degradation and reduce the reliability of flash memory. In the paper, we will propose an unclustered sorting that considers flash-memory characteristics. According to the experimental results, the improve ratio of the execution time and the energy consumption can be about 25%.

show abstract

Compression techniques for fast external sorting

Cited by 10 publications

References 29 publications

Two-way replacement selection

Two-way replacement selection

Sorting improves word-aligned bitmap indexes

Data sorting in flash memory

Contact Info

Product

Resources

About