The eXtreme-DataCloud project solutions for data management services in distributed e-infrastructures

Abstract

The eXtreme DataCloud (XDC) project is aimed at developing data management services capable to cope with very large data resources allowing the future e-infrastructures to address the needs of the next generation extreme scale scientific experiments. Started in November 2017, XDC is combining the expertise of 8 large European research organisations. The project aims at developing scalable technologies for federating storage resources and managing data in highly distributed computing environments. The project is use case driven with a multidisciplinary approach, addressing requirements from research communities belonging to a wide range of scientific domains: Life Science, Biodiversity, Clinical Research, Astrophysics, High Energy Physics and Photon Science, that represent an indicator in terms of data management needs in Europe and worldwide. The use cases proposed by the different user communities are addressed integrating different data management services ready to manage an increasing volume of data. Different scalability and performance tests have been defined to show that the XDC services can be harmonized in different contexts and complex frameworks like the European Open Science Cloud. The use cases have been used to measure the success of the project and to prove that the developments fulfil the defined needs and satisfy the final users. The present contribution describes the results carried out from the adoption of the XDC solutions and provides a complete overview of the project achievements.

Data Management of Scientific Applications in a Reinforcement Learning-Based Hierarchical Storage System

“…Such projects generate massive datasets and require complex workflows for multi-step analyses. To address these challenges, several frameworks have been introduced, such as the eXtreme DataCloud (XDC) project (Cesini et al, 2020). The project aimed at developing scalable technologies for federating storage resources.…”

Increasing data availability and fault tolerance for decentralized collaborative data-sharing systems

“…High data availability and safety is usually a requisite in order to perform large scale computations. For instance, the eXtreme-DataCloud project [1] developed smart orchestration tools and building blocks of software, which provide storage federation with transparent or quasi-transparent data access for large and geographically distributed datasets. This project was dedicated to prominent research communities from various domains: LifeScience, Biodiversity, Clinical Research, Astrophysics, High Energy Physics, and Photon Science.…”