A high-throughput big-data orchestration and processing system for the High Energy Photon Source

Abstract

High-data-throughput and multimodal-acquisition experiments will prevail in next-generation synchrotron beamlines. Orchestrating dataflow pipelines connecting the data acquisition, processing, visualization and storage ends are becoming increasingly complex and essential for enhancing beamline performance. Mamba Data Worker (MDW) has been developed to address the data challenges for the forthcoming High Energy Photon Source (HEPS). It is an important component of the Mamba experimental control and data acquisition software ecosystem, which enables fast data acquisition and transmission, dynamic configuration of data processing pipelines, data multiplex in streaming, and customized data and metadata assembly. This paper presents the architecture and development plan of MDW, outlines the essential technologies involved, and illustrates its current application at the Beijing Synchrotron Radiation Facility (BSRF).

Towards full-stack deep learning-empowered data processing pipeline for synchrotron tomography experiments

“…The boundary of tomography is pushed further with the commissioning of large scientific research facilities, such as new-generation synchrotron light sources, 19 , 20 , 21 which greatly improves the ability to characterize matter at extreme spatial and temporal resolution. This improvement is mostly due to the ultra-bright, time-resolved, and highly coherent nature of X-ray sources generated in the beamlines of new-generation synchrotron facilities, which enables synchrotron tomography.…”