Investigating interoperability of the LSST data management software stack with Astropy

Abstract

The Large Synoptic Survey Telescope (LSST) will be an 8.4 m optical survey telescope sited in Chile and capable of imaging the entire sky twice a week. The data rate of approximately 15 TB per night and the requirements to both issue alerts on transient sources within 60 seconds of observing and create annual data releases means that automated data management systems and data processing pipelines are a key deliverable of the LSST construction project. The LSST data management software has been in development since 2004 and is based on a C++ core with a Python control layer. The software consists of nearly a quarter of a million lines of code covering the system from fundamental WCS and table libraries to pipeline environments and distributed process execution.The Astropy project began in 2011 as an attempt to bring together disparate open source Python projects and build a core standard infrastructure that can be used and built upon by the astronomy community. This project has been phenomenally successful in the years since it has begun and has grown to be the de facto standard for Python software in astronomy. Astropy brings with it considerable expectations from the community on how astronomy Python software should be developed and it is clear that by the time LSST is fully operational in the 2020s many of the prospective users of the LSST software stack will expect it to be fully interoperable with Astropy.In this paper we describe the overlap between the LSST science pipeline software and Astropy software and investigate areas where the LSST software provides new functionality. We also discuss the possibilities of re-engineering the LSST science pipeline software to build upon Astropy, including the option of contributing affiliated packages.

LSST: From Science Drivers to Reference Design and Anticipated Data Products

“…The LSST Software Stack is the data processing and analysis system developed by the LSST Project to enable LSST survey data reduction and delivery. It comprises all science pipelines needed to accomplish LSST data processing tasks (e.g., calibration, single-frame processing, co-addition, image differencing, multi-epoch measurement, and asteroid orbit determination; see Bosch et al 2019 for an overview), the necessary data access (e.g., Jenness et al 2019) and orchestration middleware, and the database and user interface components. Algorithm development for the LSST software builds on the expertise and experience of prior large astronomical surveys (including SDSS, Pan-STARRS, DES, SuperMACHO, ESSENCE, DLS, CFHTLS, and UKIDSS).…”

“…The LSST Science Platform (Jurić et al 2017) represents LSST's vision for a large-scale astronomical data archive that can enable effective research with data sets of LSST size and complexity. It builds on recent trends in remote data analysis and practical experiences in the astronomical context gathered by projects such as the JHU SciServer (Raddick et al 2017), Gaia GAVIP (Vagg et al 2016), or NOAO Datalab (Fitzpatrick et al 2016).…”

The LSST Real-Time Active Optics System

“…Finally, LSST has a large central obstruction of 60 %. More information on the method used to compensate for these particular effects can be found in Xin et al 11,12 After being processed by the Wavefront Data Collector (WF Data Collector), the images are corrected from basic instrument signature using a procedure called the Instrumentation Signature Removal (ISR) 13,14 developed by the Data Management team. The ISR algorithm includes typical astronomical calibration (Flat fielding, bad pixels, darks, biases, gains).…”

Gammapy - A prototype for the CTA science tools

“…Some prominent examples are the Hubble space telescope (HST) [16], the upcoming James Webb Space Telescope (JWST) [17] and the Chandra X-ray observatory [2,18]. Even projects like LSST that started their analysis software developments before Astropy existed and are based on C++/SWIG are now actively working towards making their software interoperable with Numpy and Astropy, to avoid duplication of code and development efforts, but also to reduce the learning curve for their science tool software (since many astronomers already are using Python, Numpy and Astropy) [19].…”