Astronomical imagery: Considerations for a contemporary approach with JPEG2000

Kitaeff, Vyacheslav V.; Cannon, Andrew D.; Wicenec, A.; Taubman, David

doi:10.1016/j.ascom.2014.06.002

Cited by 23 publications

(17 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It should also be possible to stream the data progressively to the end-user, displaying an image as soon as the first data become available. Kitaeff et al (2015) and Peters and Kitaeff (2014) demonstrate the applicability and effectiveness of such approach on radio astronomy imagery.…”

Section: Streaming Imaging Datamentioning

confidence: 93%

Learning from FITS: Limitations in use in modern astronomical research

Thomas¹,

Jenness

Economou³

et al. 2015

Astronomy and Computing

View full text Add to dashboard Cite

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships.There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.

show abstract

Section: Streaming Imaging Datamentioning

confidence: 93%

Learning from FITS: Limitations in use in modern astronomical research

Thomas¹,

Jenness

Economou³

et al. 2015

Astronomy and Computing

View full text Add to dashboard Cite

show abstract

“…where size o is size of the original file and size c the size of the compressed file. Recent investigations of lossy JPEG2000 compression for astronomical images (Peters & Kitaeff, 2014;Kitaeff et al, 2015;Vohl et al, 2015) show that it can lend high factors of compression while preserving scientifically important information in the data. For example, Peters & Kitaeff (2014) compressed synthetic radio astronomy data at several levels of compression, and evaluated how the loss affects the process of source finding.…”

Section: Jpeg2000 and Lossy Data Compressionmentioning

confidence: 99%

“…This is done via the post-compression rate allocation, in which the compressed blocks are passed over to the Rate Control unit. The unit determines how many bits of the embedded bit-stream of each block should be truncated to achieve the target bit rate -aiming to minimise distortion while still reaching the target bit-rate (Kitaeff et al, 2015). Peters & Kitaeff (2014) show that the code block size and precincts size had no effect on both compression and soundness of their spectral cube data.…”

Section: Software Design Rationalementioning

confidence: 99%

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Vohl

Andreoni

Cooke

et al. 2017

Publ. Astron. Soc. Aust.

View full text Add to dashboard Cite

We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport data format to enable rapid remote searches for fast transient events as part of the Deeper Wider Faster program (DWF). DWF uses ∼20 telescopes from radio to gamma-rays to perform simultaneous and rapid-response follow-up searches for fast transient events on millisecond-to-hours timescales. DWF search demands have a set of constraints that is becoming common amongst large collaborations. Here, we focus on the rapid optical data component of DWF led by the Dark Energy Camera (DECam) at Cerro Tololo Inter-American Observatory (CTIO). Each DECam image has 70 total coupled-charged devices saved as a ∼1.2 gigabyte FITS file. Near real-time data processing and fast transient candidate identifications -in minutes for rapid follow-up triggers on other telescopes -requires computational power exceeding what is currently available on-site at CTIO. In this context, data files need to be transmitted rapidly to a foreign location for supercomputing post-processing, source finding, visualization and analysis. This step in the search process poses a major bottleneck, and reducing the data size helps accommodate faster data transmission. To maximise our gain in transfer time and still achieve our science goals, we opt for lossy data compression -keeping in mind that raw data is archived and can be evaluated at a later time. We evaluate how lossy JPEG2000 compression affects the process of finding transients, and find only a negligible effect for compression ratios up to ∼25:1. We also find a linear relation between compression ratio and the mean estimated data transmission speed-up factor. Adding highly customized compression and decompression steps to the science pipeline considerably reduces the transmission time -validating its introduction to the DWF science pipeline and enabling science that was otherwise too difficult with current technology.

show abstract

“…The astronomical community's most widespread data format, FITS [1] is 35 years old, and interest in developing a new and improved standards for formatting the larger and more varied types of astronomical data being produced by more and more complicated instruments on larger and larger telescopes is spreading [2] and [3]. Several existing options are being proposed: HDF5, a Hierarchical Data System [4], and JPEG2000, a widely-used image format [5], among others. The problems we face are not all new, and I would like to cover some history about how we got where we are and how our present solutions developed.…”

Section: Where We Arementioning

confidence: 99%

Astronomical data formats: What we have and how we got here

Mink

2015

Astronomy and Computing

View full text Add to dashboard Cite

a b s t r a c tDespite almost all being acquired as photons, astronomical data from different instruments and at different stages in its life may exist in different formats to serve different purposes. Beyond the data itself, descriptive information is associated with it as metadata, either included in the data format or in a larger multi-format data structure. Those formats may be used for the acquisition, processing, exchange, and archiving of data. It has been useful to use similar formats, or even a single standard to ease interaction with data in its various stages using familiar tools. Knowledge of the evolution and advantages of present standards is useful before we discuss the future of how astronomical data is formatted. The evolution of the use of world coordinates in FITS is presented as an example.Published by Elsevier B.V.

show abstract

Astronomical imagery: Considerations for a contemporary approach with JPEG2000

Cited by 23 publications

References 26 publications

Learning from FITS: Limitations in use in modern astronomical research

Learning from FITS: Limitations in use in modern astronomical research

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

Astronomical data formats: What we have and how we got here

Contact Info

Product

Resources

About