Cataloging the Visible Universe Through Bayesian Inference at Petascale

Abstract

Astronomical catalogs derived from wide-field imaging surveys are an important tool for understanding the Universe. We construct an astronomical catalog from 55 TB of imaging data using Celeste, a Bayesian variational inference code written entirely in the high-productivity programming language Julia. Using over 1.3 million threads on 650,000 Intel Xeon Phi cores of the Cori Phase II supercomputer, Celeste achieves a peak rate of 1.54 DP PFLOP/s. Celeste is able to jointly optimize parameters for 188M stars and galaxies, loading and processing 178 TB across 8192 nodes in 14.6 minutes. To achieve this, Celeste exploits parallelism at multiple levels (cluster, node, and thread) and accelerates I/O through Cori's Burst Buffer. Julia's native performance enables Celeste to employ high-level constructs without resorting to handwritten or generated low-level code (C/C++/Fortran), and yet achieve petascale performance.

Towards new solutions for scientific computing: the case of Julia

“…In this case, Julia codes can be as performant as other codes written using multiple libraries and languages: the typical case uses Python for most of the code and some optimized library (Numba, Fortran codes wrapped using f2py) for the most performance-critical routines. As an application of this use case we mention the Celeste project, which was able to load and process 178 TB of data from the SDSS catalogue in 14.6 minutes across 8192 nodes (Regier et al 2018). • Existing codes are monolithic and difficult to use interactively, and the expense of rewriting code in Julia can be rewarded by the possibility to run the code interactively, either in Julia's command line or in Jupyter notebooks.…”

Performance of Julia for High Energy Physics Analyses

“…The runtime supports distributed parallel as well as multi-threaded execution. It has been demonstrated to perform at peta-scale on a high-performance computing platform [17], and it has strong support for scientific machine learning [18,19,20,21]. The language implementation is open source, available under the MIT license.…”

A Probabilistic Machine Learning Approach to Scheduling Parallel Loops with Bayesian Optimization

“…L OOP parallelization is the de-facto standard method for performing shared-memory data-parallel computation. Parallel computing frameworks such as OpenMP [1] have enabled the acceleration of advances in many scientific and engineering fields such as astronomical physics [2], climate analytics [3], and machine learning [4]. A major challenge in enabling efficient loop parallelization is to deal with the inherent imbalance in workloads [5].…”