Daisy: Data analysis integrated software system for X-ray experiments

Abstract

Daisy (Data Analysis Integrated Software System) has been designed for the analysis and visualisation of X-ray experiments. To address the requirements of the Chinese radiation facilities community, spanning an extensive range from purely algorithmic problems to scientific computing infrastructure, Daisy sets up a cloud-native platform to support on-site data analysis services with fast feedback and interaction. Furthermore, the plug-in based application is convenient to process the expected high throughput data flow in parallel at next-generation facilities such as the High Energy Photon Source (HEPS). The objectives, functionality and architecture of Daisy are described in this article.

A High-Speed Asynchronous Data I/O Method for HEPS

“…Leveraging distributed memory management, it adeptly oversees the management of large-scale raw data and enhances the I/O speed for various data formats. In the future, this approach will be seamlessly integrated into Daisy [7], a purpose-built software framework tailored for data processing within HEPS. The overarching objective is to propel the acceleration of scientific computing in the future.…”

Mamba: a systematic software solution for beamline experiments at HEPS

“…Moreover, instead of mainly supporting linear processing pipelines, MDW will support full-fledged graphs of data pipes (Fig. 5), perhaps implemented in cooperation with the Daisy project (Hu et al, 2021b); this is crucial for the Figure 5 A partial data-pipe graph for a simple high-throughput tomography experiment.…”

Mamba: a systematic software solution for beamline experiments at HEPS

“…The same requirements are to our knowledge not something actively pursued by Bluesky's databroker, aside from the problem we notice that databroker is not performant enough when there is heavy disk I/O on the same machine. Moreover, instead of mainly supporting linear processing pipelines, MDW will support full-fledged graphs of data pipes (Figure 5), perhaps implemented in cooperation with the Daisy project (Hu et al, 2021b); this is crucial for the real-time tuning of fly scans, and will also be very helpful in complex multimodal experiments. As has also been discussed by Hu et al (2021a), aside from the diverse needs for processing of "real" data from experiments, the challenges posed by the many types of experiments possible at HEPS also include the complexity in management of the scientific metadata for these experiments, which will directly affect the formatting of raw data by MDW.…”