Learning from FITS: Limitations in use in modern astronomical research

Abstract

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships.There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.

FITS Format for Planetary Surfaces: Definitions, Applications, and Best Practices

“…We understand that some believe FITS has grown too old for newer astronomical applications (e.g., Thomas et al, ). We are also not convinced that one format can solve all the issues faced from long‐term archiving to online streaming.…”

ASDF: A new data format for astronomy

“…Nevertheless, the shortcomings of the standard astronomical data format are becoming more of a burden on those that must create and process data from new telescopes. These shortcomings are discussed in great length in a paper (Thomas et al, 2015) and will not be rehashed here. The community appears to be split on whether this means that incremental improvements to FITS are needed, or a completely different format is appropriate, but the trend of opinion toward needing an alternate format appears to be growing.…”

“…(These factors are detailed at length in Thomas et al, 2015; these include the severe restrictions on keyword name length, the lack of any simple grouping structure for metadata or data, among others. )…”

AST: A library for modelling and manipulating coordinate systems

“…WCS data in FITS data files is stored in a series of 80-character header cards and, over the years, the number of different ways the information can be stored in these headers has grown. The complexity involved is now considerable (see e.g., Thomas et al, 2015). Again, detailed specialist knowledge is needed to extract this information reliably and to write it back (possibly modified) in a form that gives other FITS-handling software a chance of using it while not conflicting with the many other FITS headers typically present.…”