dCache: Inter-disciplinary storage system

Self Cite

The ever-increasing amount of data that is produced by modern scientific facilities like EuXFEL or LHC puts high pressure on the data management infrastructure at the laboratories. This includes poorly shareable resources of archival storage, typically, tape libraries. To achieve maximal efficiency of the available tape resources, a deep integration between hardware and software components is required. The CERN Tape Archive (CTA) is an open-source storage management system developed by CERN to manage LHC experiment data on tape. Although today CTA’s primary target is CERN Tier-0, the data management group at DESY considers the CTA as a main alternative to commercial HSM systems. dCache has an extensible tape interface that allows connectivity to almost any tape system. Together with the CERN Tape Archive developers, the storage team at DESY is working on seamless integration of CTA into dCache.

Section: Cern Tape Arcivementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

dCache integration with CERN Tape Archive

Mkrtchyan,

Chodak,

Karimi

et al. 2024

Self Cite

“…Some WLCG sites would like to use CTA with dCache [20] rather than EOS. In principle, CTA is agnostic to the disk system in front.…”

Section: Choice Of Disk Systemmentioning

confidence: 99%

The CERN Tape Archive Beyond CERN An Open Source Data Archival System for HEP

Davis,

Afonso,

Bachmann

et al. 2024

The CERN Tape Archive (CTA) is the successor to CASTOR and the tape backend to EOS. It was designed to meet the needs of archival storage of data from LHC Run–3 and other experimental programmes at CERN. In the wider Worldwide LHC Computing Grid (WLCG), the tape software landscape is quite heterogeneous, but we are entering a period of consolidation. A number of sites have reevaluated their options and are choosing CTA for their future tape archival storage needs. However, CTA’s original mandate imposed several design constraints which are not necessarily optimal for external sites. In this contribution, we show how CERN has engaged with the wider HEP community and collaborated on improvements which allow CTA to be adopted more widely. We detail community contributions to allow CTA to be used as the tape backend for dCache; to facilitate migrations from other tape systems such as OSM and Enstore; and improvements to CTA building and packaging to remove CERN-specific dependencies and to allow easy distribution to external sites. Finally, we present a roadmap for the community edition of CTA.

“…dCache [16] is popular storage implementation to host and archive scientific data sets. A major fraction of all WLCG data is stored on dCache storage and for example the LOFAR Long Term Archive makes also use of dCache.…”

Section: Federated Storage Infrastructure -Storage4punchmentioning

confidence: 99%

Federated Heterogeneous Compute and Storage Infrastructure for the PUNCH4NFDI Consortium

Drabent,

Freyermuth,

Giffels

et al. 2024

PUNCH4NFDI, funded by the Germany Research Foundation initially for five years, is a diverse consortium of particle, astro-, astroparticle, hadron and nuclear physics embedded in the National Research Data Infrastructure initiative. In order to provide seamless and federated access to the huge variety of compute and storage systems provided by the participating communities covering their very diverse needs, the Compute4PUNCH and Storage4PUNCH concepts have been developed. Both concepts comprise state-of-the-art technologies such as a token-based AAI for standardized access to compute and storage resources. The community supplied heterogeneous HPC, HTC and Cloud compute resources are dynamically and transparently integrated into one federated HTCondorbased overlay batch system using the COBalD/TARDIS resource meta-scheduler. Traditional login nodes and a JupyterHub provide entry points into the entire landscape of available compute resources, while container technologies and the CERN Virtual Machine File System (CVMFS) ensure a scalable provisioning of community-specific software environments. In Storage4PUNCH, community supplied storage systems mainly based on dCache or XRootD technology are being federated in a common infrastructure employing methods that are well established in the wider HEP community. Furthermore existing technologies for caching as well as metadata handling are being evaluated with the aim for a deeper integration. The combined Compute4PUNCH and Storage4PUNCH environment will allow a large variety of researchers to carry out resource-demanding analysis tasks. In this contribution we will present the Compute4PUNCH and Storage4PUNCH concepts, the current status of the developments as well as first experiences with scientific applications being executed on the available prototypes.