Reimplementing the Hierarchical Data System using HDF5

Jenness, Tim

doi:10.1016/j.ascom.2015.02.003

Cited by 13 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As mentioned before, numerous data formats have been proposed in the recent years for archiving large datasets. If desired, cuFFS can be modified to support additional data formats like N-dimensional Data Format (NDF; Jenness et al 2015), Starlink Hierarchical Data System (HDS;Jenness 2015) and Advanced Scientific Data Format (ASDF; Greenfield et al 2015) with minimal change to the existing code.…”

Section: A Few Comments On File Formatsmentioning

confidence: 99%

cuFFS: A GPU-accelerated code for Fast Faraday rotation measure Synthesis

Sridhar

Heald

Hulst

2018

Astronomy and Computing

View full text Add to dashboard Cite

Rotation measure (RM) synthesis is a widely used polarization processing algorithm for reconstructing polarized structures along the line of sight. Performing RM synthesis on large datasets produced by telescopes like LOFAR can be computationally intensive as the computational cost is proportional to the product of the number of input frequency channels, the number of output Faraday depth values to be evaluated and the number of lines of sight present in the data cube. The required computational cost is likely to get worse due to the planned large area sky surveys with telescopes like the Low Frequency Array (LOFAR), the Murchison Widefield Array (MWA), and eventually the Square Kilometre Array (SKA). The massively parallel General Purpose Graphical Processing Units (GPGPUs) can be used to execute some of the computationally intensive astronomical image processing algorithms including RM synthesis. In this paper, we present a GPU-accelerated code, called cuFFS or CUDA-accelerated Fast Faraday Synthesis, to perform Faraday rotation measure synthesis. Compared to a fast single-threaded and vectorized CPU implementation, depending on the structure and format of the data cubes, our code achieves an increase in speed of up to two orders of magnitude. During testing, we noticed that the disk I/O when using the Flexible Image Transport System (FITS) data format is a major bottleneck and to reduce the time spent on disk I/O, our code supports the faster HDFITS format in addition to the standard FITS format. The code is written in C with GPU-acceleration achieved using Nvidia's CUDA parallel computing platform. The code is available at https://github.com/sarrvesh/cuFFS.

show abstract

Section: A Few Comments On File Formatsmentioning

confidence: 99%

cuFFS: A GPU-accelerated code for Fast Faraday rotation measure Synthesis

Sridhar

Heald

Hulst

2018

Astronomy and Computing

View full text Add to dashboard Cite

show abstract

“…Raw data files at JCMT are written to HDS format files (e.g., Jenness 2015Jenness , ascl:1502.009) using the extensible N -Dimensional Data Format (NDF; Jenness et al 2015bJenness et al , ascl:1411). The spectral data are stored in a threedimensional data array dimensioned by spectrum channels/frequency, detector number and time.…”

Section: Appendix A: Jcmt Heterodyne Raw Data Modelmentioning

confidence: 99%

Automated reduction of sub-millimetre single-dish heterodyne data from the James Clerk Maxwell Telescope using orac-dr

Jenness

Currie

Tilanus

et al. 2015

Mon. Not. R. Astron. Soc.

Self Cite

View full text Add to dashboard Cite

With the advent of modern multi-detector heterodyne instruments that can result in observations generating thousands of spectra per minute it is no longer feasible to reduce these data as individual spectra. We describe the automated data reduction procedure used to generate baselined data cubes from heterodyne data obtained at the James Clerk Maxwell Telescope. The system can automatically detect baseline regions in spectra and automatically determine regridding parameters, all without input from a user. Additionally it can detect and remove spectra suffering from transient interference effects or anomalous baselines. The pipeline is written as a set of recipes using the ORAC-DR pipeline environment with the algorithmic code using Starlink software packages and infrastructure. The algorithms presented here can be applied to other heterodyne array instruments and have been applied to data from historical JCMT heterodyne instrumentation.

show abstract

“…The astronomical community's most widespread data format, FITS [1] is 35 years old, and interest in developing a new and improved standards for formatting the larger and more varied types of astronomical data being produced by more and more complicated instruments on larger and larger telescopes is spreading [2] and [3]. Several existing options are being proposed: HDF5, a Hierarchical Data System [4], and JPEG2000, a widely-used image format [5], among others. The problems we face are not all new, and I would like to cover some history about how we got where we are and how our present solutions developed.…”

Section: Where We Arementioning

confidence: 99%

Astronomical data formats: What we have and how we got here

Mink

2015

Astronomy and Computing

View full text Add to dashboard Cite

a b s t r a c tDespite almost all being acquired as photons, astronomical data from different instruments and at different stages in its life may exist in different formats to serve different purposes. Beyond the data itself, descriptive information is associated with it as metadata, either included in the data format or in a larger multi-format data structure. Those formats may be used for the acquisition, processing, exchange, and archiving of data. It has been useful to use similar formats, or even a single standard to ease interaction with data in its various stages using familiar tools. Knowledge of the evolution and advantages of present standards is useful before we discuss the future of how astronomical data is formatted. The evolution of the use of world coordinates in FITS is presented as an example.Published by Elsevier B.V.

show abstract

Reimplementing the Hierarchical Data System using HDF5

Cited by 13 publications

References 18 publications

cuFFS: A GPU-accelerated code for Fast Faraday rotation measure Synthesis

cuFFS: A GPU-accelerated code for Fast Faraday rotation measure Synthesis

Automated reduction of sub-millimetre single-dish heterodyne data from the James Clerk Maxwell Telescope using orac-dr

Astronomical data formats: What we have and how we got here

Contact Info

Product

Resources

About