The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery

Peters, Sean M.; Kitaeff, Vyacheslav V.

doi:10.1016/j.ascom.2014.06.003

Cited by 12 publications

(16 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From these results, we estimate that utilising lossy JPEG2000 compression with a compression ratio up to 25:1 enables the DWF team to efficiently retrieve transient sources within the detection limits of the survey without significant loss. These results are in agreement with those obtained by Peters & Kitaeff (2014)…”

Section: Cross-match Listssupporting

confidence: 93%

“…where size o is size of the original file and size c the size of the compressed file. Recent investigations of lossy JPEG2000 compression for astronomical images (Peters & Kitaeff, 2014;Kitaeff et al, 2015;Vohl et al, 2015) show that it can lend high factors of compression while preserving scientifically important information in the data. For example, Peters & Kitaeff (2014) compressed synthetic radio astronomy data at several levels of compression, and evaluated how the loss affects the process of source finding.…”

Section: Jpeg2000 and Lossy Data Compressionmentioning

confidence: 99%

“…Recent investigations of lossy JPEG2000 compression for astronomical images (Peters & Kitaeff, 2014;Kitaeff et al, 2015;Vohl et al, 2015) show that it can lend high factors of compression while preserving scientifically important information in the data. For example, Peters & Kitaeff (2014) compressed synthetic radio astronomy data at several levels of compression, and evaluated how the loss affects the process of source finding. In this case it was shown that the strongest sources (2000 mJy km/s and higher) could still be retrieved at extremely high compression ratio, where the compressed file would be more than 15,000 times smaller than the original file.…”

Section: Jpeg2000 and Lossy Data Compressionmentioning

confidence: 99%

See 2 more Smart Citations

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Vohl

Andreoni

Cooke

et al. 2017

Publ. Astron. Soc. Aust.

View full text Add to dashboard Cite

We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport data format to enable rapid remote searches for fast transient events as part of the Deeper Wider Faster program (DWF). DWF uses ∼20 telescopes from radio to gamma-rays to perform simultaneous and rapid-response follow-up searches for fast transient events on millisecond-to-hours timescales. DWF search demands have a set of constraints that is becoming common amongst large collaborations. Here, we focus on the rapid optical data component of DWF led by the Dark Energy Camera (DECam) at Cerro Tololo Inter-American Observatory (CTIO). Each DECam image has 70 total coupled-charged devices saved as a ∼1.2 gigabyte FITS file. Near real-time data processing and fast transient candidate identifications -in minutes for rapid follow-up triggers on other telescopes -requires computational power exceeding what is currently available on-site at CTIO. In this context, data files need to be transmitted rapidly to a foreign location for supercomputing post-processing, source finding, visualization and analysis. This step in the search process poses a major bottleneck, and reducing the data size helps accommodate faster data transmission. To maximise our gain in transfer time and still achieve our science goals, we opt for lossy data compression -keeping in mind that raw data is archived and can be evaluated at a later time. We evaluate how lossy JPEG2000 compression affects the process of finding transients, and find only a negligible effect for compression ratios up to ∼25:1. We also find a linear relation between compression ratio and the mean estimated data transmission speed-up factor. Adding highly customized compression and decompression steps to the science pipeline considerably reduces the transmission time -validating its introduction to the DWF science pipeline and enabling science that was otherwise too difficult with current technology.

show abstract

Section: Cross-match Listssupporting

confidence: 93%

Section: Jpeg2000 and Lossy Data Compressionmentioning

confidence: 99%

Section: Jpeg2000 and Lossy Data Compressionmentioning

confidence: 99%

See 1 more Smart Citation

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Vohl

Andreoni

Cooke

et al. 2017

Publ. Astron. Soc. Aust.

View full text Add to dashboard Cite

show abstract

“…Recent work by Kitaeff et al (2012), Kitaeff et al (2014, this issue), and Peters & Kitaeff (2014) investigates whether the JPEG2000 (ISO/IEC 15444) standards could be adopted more generally within astronomy. Within the JPEG2000 specification are features attractive to astrophysics such as: progressive transmission; the ability to decode only part of the data without having to load it all into memory (useful when dealing with larger-than-memory files); and the possibility to include customized metadata.…”

Section: Big Data Data Format and Data Compressionmentioning

confidence: 99%

Data compression in the petascale astronomy era: A GERLUMPH case study

Vohl

Fluke

Vernardos

2015

Astronomy and Computing

View full text Add to dashboard Cite

As the volume of data grows, astronomers are increasingly faced with choices on what data to keep -and what to throw away. Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has shown promising results on observational data. However, there is still a need to evaluate its potential on other type of astronomical data, such as from numerical simulations. GERLUMPH (the GPU-Enabled High Resolution cosmological MicroLensing parameter survey) represents an example of a data intensive project in theoretical astrophysics. In the next phase of processing, the ≈ 27 terabyte GERLUMPH dataset is set to grow by a factor of 100 -well beyond the current storage capabilities of the supercomputing facility on which it resides. In order to minimise bandwidth usage, file transfer time, and storage space, this work evaluates several data compression techniques. Specifically, we investigate off-the-shelf and custom lossless compression algorithms as well as the lossy JPEG2000 compression format. Results of lossless compression algorithms on GERLUMPH data products show small compression ratios (1.35:1 to 4.69:1 of input file size) varying with the nature of the input data. Our results suggest that JPEG2000 could be suitable for other numerical datasets stored as gridded data or volumetric data. When approaching lossy data compression, one should keep in mind the intended purposes of the data to be compressed, and evaluate the effect of the loss on future analysis. In our case study, lossy compression and a high compression ratio do not significantly compromise the intended use of the data for constraining quasar source profiles from cosmological microlensing.

show abstract

“…18 http://www.openjpeg.org.19 v2.0 was already available at the time when the paper was written. 20 f2j does not transfer FITS headers to JPEG2000 files, however, the software described inPeters and Kitaeff (2014) does transfer FITS headers into metadata boxes of JPX. 21 https://github.com/ICRAR/f2j.git.…”

mentioning

confidence: 98%

Astronomical imagery: Considerations for a contemporary approach with JPEG2000

Kitaeff

Cannon

Wicenec

et al. 2015

Astronomy and Computing

Self Cite

View full text Add to dashboard Cite

a b s t r a c tThe new wide-field radio telescopes, such as: ASKAP, MWA, LOFAR, eVLA and SKA; will produce spectralimaging data-cubes (SIDC) of unprecedented size-in the order of hundreds of Petabytes. Servicing such data as images to the end-user in a traditional manner and formats is likely going to encounter significant performance fallbacks. We discuss the requirements for extremely large SIDCs, and in this light we analyse the applicability of the approach taken in the JPEG2000 (ISO/IEC 15444) standards. We argue the case for the adaptation of contemporary industry standards and technologies versus the modification of legacy astronomy standards or development of new standards specific to astronomy only.

show abstract

The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery

Cited by 12 publications

References 21 publications

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Data compression in the petascale astronomy era: A GERLUMPH case study

Astronomical imagery: Considerations for a contemporary approach with JPEG2000

Contact Info

Product

Resources

About