A Lossless Astronomical Data Compression Scheme with FPGA Acceleration

Zheng, Yu; Zhu, Yongxin; Song, Yibing; Nan, Tianhao; Li, Wanyi

doi:10.1109/socc46988.2019.1570562786

Cited by 3 publications

(5 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparing compression speeds for the SKA dataset, with the results obtained by Zheng et al [49] and Masui et al [46], is not appropriate as they use hardware configurations that differ greatly from that used here. Masui et al execute their tests on data files stored entirely in memory, removing any I/O overhead of a permanent storage medium such as an SSD [46].…”

Section: Compression and Decompression Speedsmentioning

confidence: 85%

“…If further work is conducted on reordering radio-astronomy data for the purpose of compression, the description vector is an appropriate and necessary tool. The compression ratios achieved are good, but comparing these results to those in the literature indicates that there may be other compression schemes that are better suited for the datasets with SFC reorderings [42], [45], [46], [49]. Applying the following methodology to other compression schemes would expand upon the knowledge of how such reorderings effect compression performance.…”

Section: Future Workmentioning

confidence: 95%

“…But it does highlight that radio astronomy data is far more redundant when some precision is lost for the sake of size requirements. The work by Zheng et al, in compressing astronomy data using an effective bitwidth transform with GZIP, shows that there is still room for lossless compression schemes for similar datasets [49]. Even so, their dataset, and that used by Masui et al, contains integer data and not floating-point data as is the case with the SKA dataset.…”

Section: B Compression Ratiosmentioning

confidence: 99%

“…This is why they are able to achieve a write-speed of 749 MiB/s. Zheng et al do not store their data files in memory, but they do use an FPGA as an external accelerator [49]. They do experiment with executing their technique on a CPU alone, but it is nearly seven times slower than BitShuffle with LZ4.…”

Section: Compression and Decompression Speedsmentioning

confidence: 99%

“…Though they apply the technique to integer data, it is also applicable to floating-point data. Zheng et al implement a lossless compression technique for integer astronomy data also by reformatting low-level bit representations of values [49]. Instead of grouping bits by their significance, an effective bit-width is used alongside DEFLATE to achieve better compression than the combination of BitShuffle and LZ4 (R c = 3.36 vs R c = 2.49) on an integer dataset from the Five Hundred Meter Aperture Spherical Radio Telescope (FAST) [48].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Effect of d-Dimensional Re-orderings on Lossless Compression of Radio-Astronomy and Digital Elevation Data

2021

View full text Add to dashboard Cite

For multidimensional data, Space-Filling Curves (SFCs) have been used to improve the execution time of spatial data queries. However, their effect on compression, when used to reorder the uncompressed values, is known to a lesser extent. We investigate the impact of three SFCs on Shuttle Radar Topographic Mission (SRTM) elevation data and Square-Kilometre Array telescope (SKA) radio-astronomy data: two types of datasets to which SFCs have not been extensively applied, within a compression context. This work contributes to the understanding of how such reorderings impact compression performance and affect different compression schemes and preprocessing techniques through their use. We show empirical results from combining eight common compression schemes, the Z-Order, Gray-Code, and Hilbert spacefilling curves, and the bitwise preprocessing technique BitShuffle. The Hilbert Curve consistently outperforms the other orderings for the SRTM dataset though the mapping implementation incurs a significant speed penalty. However, the Z-Order and Gray-Code Curves are best for the SKA dataset. Through an analysis of the dataset autocorrelations, file-entropies, and block-entropies; we show that the SKA dataset's dimensional bias is not exploited as much by the Hilbert Curve compared to the Z-Order and Gray-Code Curves. However, the Hilbert Curve is the most appropriate for the SRTM dataset as it can be modelled as isotropic and has a significantly higher level of local autocorrelation. BitShuffle is necessary to practically compress the SKA data, but does contribute to the compression performance of the SRTM dataset. These curves and BitShuffle are advantageous in reducing block-entropy values for such datasets.

show abstract

Section: Compression and Decompression Speedsmentioning

confidence: 85%