The ESCAPE Data Lake: The machinery behind testing, monitoring and supporting a unified federated storage infrastructure of the exabyte-scale

Dona, Rizart; Maria, R. Di; Escape,

doi:10.1051/epjconf/202125102060

Cited by 2 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another benefit is that FTS is adopted worldwide in various data centres, presenting collaboration opportunities with Fenix. We show such potential in Figure 1 as extended data transferability to the ESCAPE Data Lake [6].…”

Section: Introductionmentioning

confidence: 82%

“…Basic functions such as uploading, downloading and deleting files (objects in cloud terminologies) as well as advanced requests with checksum, stat and partial read are supported. We also enabled the Swift Static Large Object (SLO) 6 support for DaviX, namely, large files are uploaded in segments to the Swift object store automatically to provide efficiency.…”

Section: File Management With Davixmentioning

confidence: 99%

“…It is a model that can be extended with external platforms, providing a more powerful framework. Our attention is directed towards the ESCAPE project, with a specific focus on enabling data transfer between the ESCAPE Data Lake [6] and a Fenix ARD.…”

Section: Proof-of-concept Setupmentioning

confidence: 99%

“…One step towards enabling the processing of experimental data within Fenix is the realisation of a data transfer service that allows moving experimental data to the relevant supercomputing centres and vice versa. More specifically, we consider the case where the experimental data is stored in the ESCAPE Data Lake [6]. For this purpose, we enhanced the File Transfer Service (FTS), specifically its current iteration FTS3 [7] to support Fenix storage services.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Integrating FTS in the Fenix HPC Infrastructure

Long,

Pleiter,

Patrascoiu

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

As compute requirements in experimental high-energy physics are expected to significantly increase, there is a need for leveraging high-performance computing (HPC) resources. However, HPC systems are currently organised and operated in a way that this is not easily possible. Here we will focus on a specific e-infrastructure that incorporates HPC resources, namely Fenix, which is based on a consortium of 6 leading European supercomputing centres. Fenix was initiated through the Human Brain Project (HBP) but also provides resources to other research communities in Europe. The Fenix sites are integrated into a common AAI and provide a so-called Archival Data Repository that can be accessed through a Swift API. In this paper, we report on our efforts to realise a data transfer service that allow to exchange data with the Fenix e-infrastructure. This has been enabled by implementing support of Swift in FTS3 and related software components. We will, in particular, discuss how FTS3 has been integrated into the Fenix AAI, which largely follows the architectural principles of the European Open Science Cloud (EOSC). Furthermore, we show how end-users can use this service through a WebFTS service that has been integrated into the science gateway of the HBP, which is also known as the HBP Collaboratory. Finally, we discuss how transfer commands can be automatically distributed over several FTS3 instances to optimise transfer between different Fenix sites.

show abstract

Section: Introductionmentioning

confidence: 82%

Section: File Management With Davixmentioning

confidence: 99%

Section: Proof-of-concept Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Integrating FTS in the Fenix HPC Infrastructure

Long,

Pleiter,

Patrascoiu

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

show abstract

A Review of Technologies and Challenges for Integrated Modeling Analysis

Sigler,

Ubach,

Mora

et al. 2024

Arch Computat Methods Eng

View full text Add to dashboard Cite

The natural and built environment form a complex system, comprised of many interrelated subsystems, each interacting in multiple nexus. Manifestations of these interactions can be seen in complex events. Climate change, natural disasters, military conflicts, pandemics, and other such events require accurate preparation, preparedness, and response planning, in a fast, ever changing context. With exascale ($$10^{18}$$ 10 18 floating point operations per second) computational levels reached, computing power gives us the capability to model and simulate complex scenarios. This capability gives decision makers tools to game possibilities and enact preparatory and preventative measures to build resilience. There has been a trend of development of decision support, risk assessment, and operational forecasting systems to address this issue, aggregating diverse data sources onto unified platforms. Nonetheless, the majority of such platforms focus on the aggregation of just data and not models, and remain in silos of disciplines. What is needed to prepare and plan for disruptive events is a move towards decision support based on holistic, integrated, model-based analysis. While modeling individual systems has been done for many years, modeling in holistic analysis presents additional challenges. This paper presents an overview of the challenges and advances present for a move to a model-based holistic analysis, and an evaluation of some platforms currently in development and operation. The present work signals gaps in research to be addressed. Finally, we formulate base requirements for the development of systems to perform holistic model-based analysis, and discuss future plans for the development of such a platform.

show abstract

The ESCAPE Data Lake: The machinery behind testing, monitoring and supporting a unified federated storage infrastructure of the exabyte-scale

Cited by 2 publications

References 10 publications

Integrating FTS in the Fenix HPC Infrastructure

Integrating FTS in the Fenix HPC Infrastructure

A Review of Technologies and Challenges for Integrated Modeling Analysis

Contact Info

Product

Resources

About