ASDF: A new data format for astronomy

Greenfield, P.; Droettboom, Michael; Bray, Erik M.

doi:10.1016/j.ascom.2015.06.004

Cited by 49 publications

(42 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the list below, we have summarized the limitations and challenges from Greenfield et al (2015) that are most relevant to neuroscience along with some additional drawbacks which are addressed with Exdir:…”

Section: Existing Alternativesmentioning

confidence: 99%

“…2.1 Hierarchical Data Format (HDF5) HDF5 holds many advantages over alternative data formats, but as described by Greenfield et al (2015), the HDF5 format also has crucial disadvantages. In the list below, we have summarized the limitations and challenges from Greenfield et al (2015) that are most relevant to neuroscience along with some additional drawbacks which are addressed with Exdir:…”

Section: Existing Alternativesmentioning

confidence: 99%

“…2.2 Advanced Scientific Data Format (ASDF) Greenfield et al (2015) propose a new format (Advanced Scientific Data Format, ASDF) to address many of the above mentioned problems. Similar to Exdir, ASDF also embraces YAML for metadata, but it also stores and organizes binary data in the same YAML file.…”

Section: Existing Alternativesmentioning

confidence: 99%

See 2 more Smart Citations

Experimental Directory Structure (Exdir): An alternative to HDF5 without introducing a new file format

Dragly

Mobarhan

Lepperød

et al. 2018

Preprint

View full text Add to dashboard Cite

Natural sciences generate an increasing amount of data in a wide range of formats developed by different research groups and commercial companies. At the same time there is a growing desire to share data along with publications in order to enable reproducible research. Open formats have publicly available specifications which facilitate data sharing and reproducible research. Hierarchical Data Format 5 (HDF5) is a popular open format widely used in neuroscience, often as a foundation for other, more specialized formats. However, drawbacks related to HDF5's complex specification have initiated a discussion for an improved replacement. We propose a novel alternative, the Experimental Directory Structure (Exdir), an open standard for data storage in experimental pipelines which amends drawbacks associated with HDF5 while retaining its advantages. HDF5 stores data and metadata in a hierarchy within a complex binary file which, among other things, is not human-readable, not optimal for version control systems, and lacks support for storing raw data. Exdir, one the other hand, uses file system directories to represent the hierarchy, with metadata stored in human-readable YAML files, datasets stored in binary NumPy files, and raw data stored directly in subdirectories. Furthermore, storing data in multiple files makes it easier to track for version control systems. Exdir is not a file format in itself, but a standard for organizing files in a directory structure. Exdir uses the same abstractions as HDF5 and is compatible with the HDF5 Abstract Data Model. Several research groups are already using data stored in a directory hierarchy as an alternative to HDF5, but no common standard exists in the field. This complicates and limits the opportunity for data sharing and development of common tools for reading, writing, and analyzing data. Exdir facilitates improved data storage, data sharing, reproducible research, and novel insight from interdisciplinary collaboration. With the publication of Exdir, we invite the scientific community to join the development to create an open standard that will serve as many needs as possible and that will serve as a foundation for open access to and exchange of data.

show abstract

Section: Existing Alternativesmentioning

confidence: 99%

Section: Existing Alternativesmentioning

confidence: 99%

See 1 more Smart Citation

Experimental Directory Structure (Exdir): An alternative to HDF5 without introducing a new file format

Dragly

Mobarhan

Lepperød

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…GWCS can serialize to FITS-WCS and also into YAML 44 using the Advanced Scientific Data Format (ASDF). 45 We have decided to investigate a common format that is based on YAML and the proposed new variant of STC, version 2.0. STC-2 46 is an evolution of STC that includes support for chainable mappings and seems like it could be a suitable starting point for this investigation.…”

Section: Generalized World Coordinate Systemsmentioning

confidence: 99%

Investigating interoperability of the LSST data management software stack with Astropy

et al. 2016

View full text Add to dashboard Cite

The Large Synoptic Survey Telescope (LSST) will be an 8.4 m optical survey telescope sited in Chile and capable of imaging the entire sky twice a week. The data rate of approximately 15 TB per night and the requirements to both issue alerts on transient sources within 60 seconds of observing and create annual data releases means that automated data management systems and data processing pipelines are a key deliverable of the LSST construction project. The LSST data management software has been in development since 2004 and is based on a C++ core with a Python control layer. The software consists of nearly a quarter of a million lines of code covering the system from fundamental WCS and table libraries to pipeline environments and distributed process execution.The Astropy project began in 2011 as an attempt to bring together disparate open source Python projects and build a core standard infrastructure that can be used and built upon by the astronomy community. This project has been phenomenally successful in the years since it has begun and has grown to be the de facto standard for Python software in astronomy. Astropy brings with it considerable expectations from the community on how astronomy Python software should be developed and it is clear that by the time LSST is fully operational in the 2020s many of the prospective users of the LSST software stack will expect it to be fully interoperable with Astropy.In this paper we describe the overlap between the LSST science pipeline software and Astropy software and investigate areas where the LSST software provides new functionality. We also discuss the possibilities of re-engineering the LSST science pipeline software to build upon Astropy, including the option of contributing affiliated packages.

show abstract

“…Interest in flexible schemes for representing inter-related coordinate systems has increased recently -for instance, in the discussions about possible successors to the FITS format (Greenfield et al, 2015;Mink et al, 2015;Shortridge, 2016), and within the Astropy (Astropy Collaboration, 2013; Dencheva et al, 2016) and International Virtual Observatory Alliance (IVOA) (STC2; Rots, in preparation) projects. These discussions suggest it is an appropriate time to review the lessons learned from AST.…”

Section: Introductionmentioning

confidence: 99%

AST: A library for modelling and manipulating coordinate systems

Berry

Warren-Smith

Jenness

2016

Astronomy and Computing

View full text Add to dashboard Cite

In view of increased interest in object-oriented systems for describing coordinate information, we present a description of the data model used by the Starlink AST library. AST provides a comprehensive range of facilities for attaching world co-ordinate systems to astronomical data, and for retrieving and interpreting that information in a variety of formats, including FITS-WCS. AST is a mature system that has been in use for more than 17 years, and may consequently be useful as a means of informing development of similar systems in the future.

show abstract

ASDF: A new data format for astronomy

Cited by 49 publications

References 18 publications

Experimental Directory Structure (Exdir): An alternative to HDF5 without introducing a new file format

Experimental Directory Structure (Exdir): An alternative to HDF5 without introducing a new file format

Investigating interoperability of the LSST data management software stack with Astropy

AST: A library for modelling and manipulating coordinate systems

Contact Info

Product

Resources

About