Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis

Abstract

SummaryX‐ray scattering experiments using free electron lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID‐19 viral proteins). XFEL experiments are a challenge to computing in two ways: (i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; (ii) data‐collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min—sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS‐NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Completing data analysis within 10 min is a first for XFEL experiments and an important milestone if we are to keep up with data‐collection trends.

A High-level Design for Bidirectional Data Streaming to High-Performance Computing Systems from External Science Facilities

“…The Linac Coherent Light Source (LCLS) at SLAC National Accelerator Laboratory provides X-ray scattering for the study of the molecular structure and function properties of materials and substances. Past work [2] has demonstrated a scientific workflow that forwards experiment data from LCLS to the National Energy Research Scientific Center (NERSC) at Lawrence Berkeley National Laboratory (LBL) to provide quick data analysis that enables scientists to make informed decisions when configuring subsequent experiment runs. The existing workflow uses file-based transfers of experimental image data from the SLAC analysis file system to NERSC shared storage systems, using a clustered data transfer service known as XRootD that runs at both sites.…”

“…For this capability we want to focus on memory-to-memory data movement (i.e., moving data directly from the memory of producer applications to the memory of consumer applications) and avoid requiring data transfers between file systems as the basis for moving data between experimental and HPC systems. File-based data movement is already well-supported by the U. S. Department of Energy (DOE) Advanced Scientific Computing Research (ASCR) user facilities and has been shown to be a bottleneck for certain use cases [2,20].…”