Third-party transfers in WLCG using HTTP

Bockelman, Brian; Ceccanti, Andrea; Furano, Fabrizio; Millar, Paul; Litvintsev, D.; Forti, A.

doi:10.1051/epjconf/202024504031

Cited by 5 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The process that led to the creation of the HTTP Tape REST API started around 2018, with a movement in the WLCG community to migrate disk-to-disk transfers to HTTP-TPC (HTTP third-party copy). At the time, most WLCG Tier-1 transfers were using GridFTP, which had to be deprecated (more details about this can be found on [15]). While most of WLCG Tier-1 disk storage endpoints moved to HTTP-TPC early-on, the disk part of tape-endpoints had to remain unchanged for a while waiting for extra development work (due to HTTP not supporting tape operations).…”

Section: Previous Workmentioning

confidence: 99%

An HTTP REST API for Tape-backed Storage

Afonso,

Caffy,

Patrascoiu

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

The goal of the HTTP REST API for Tape project is to provide a simple, minimalistic and uniform interface to manage data transfers between Storage Endpoints (SEs) where the source file is on tape. The project is a collaboration between the developers of WLCG storage systems (EOS+CTA, dCache, StoRM) and data transfer clients (gfal2, FTS). For some years, HTTP has been growing in popularity as the preferred data transfer protocol between many WLCG SEs. However — unlike other protocols such as XRootD and SRM — HTTP does not include a method to stage files from tape to disk prior to transfer, forcing the use of workarounds such as hybrid protocols (different protocols used for the “stage” and “transfer” parts of the operation). The HTTP REST API offers a simple and consistent solution, by extending the HTTP protocol to include staging operations. It provides clients with a common and consistent API across different storage systems to manage and monitor the disk and tape residency of stored files. In this contribution, we present the history and motivation of the HTTP REST API project, the specification of version 1 of the API and implementation details in the various storage and data transfer systems. We also describe our experience of its deployment and use for LHC Run-3 operations. We conclude with a discussion of possible future work.

show abstract

Section: Previous Workmentioning

confidence: 99%

An HTTP REST API for Tape-backed Storage

Afonso,

Caffy,

Patrascoiu

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

show abstract

“…To As a consequence, the collaboration decided to increase the number of concurrent transfers and also the files' size up to 100 MB, checking both pull and push mode [11] [12] with gfal-copy.…”

Section: Third-party Copiesmentioning

confidence: 99%

CNAF experience in support of the JUNO distributed computing model

Rendina,

Morganti,

Fornari

et al. 2023

Proceedings of International Symposium on Grids &Amp; Clouds (ISGC) 2023 in Conjunction With HEPiX Spring 2023 Workshop — PoS(I

View full text Add to dashboard Cite

The Italian WLCG Tier-1 located in Bologna and managed by INFN CNAF provides computing and storage resources to several research communities in the fields of High-Energy Physics, Astroparticle Physics, Gravitational Waves, Nuclear Physics and others. Among them, the Jiangmen Underground Neutrino Observatory (JUNO), devoted to the construction and operation of a neutrino detector located underground in Kaiping, Jiangmen in Southern China, will employ a computing infrastructure geographically distributed in Chinese, Russian, French and Italian data centers. The detector data rate is expected to be of the order of 2 PB per year, continuously transferred from the detector site to the INFN Tier-1 in Italy. To guarantee the optimal operations among all the aforementioned sites, a series of periodic network and data management challenges has been performed. In this talk, the technologies involved to set up the cross-continent data transfer (e.g. StoRM WebDAV, EOS, dCache, FTS, Rucio) are presented, together with their performance.

show abstract

“…Additionally, we discuss the validation and commissioning tests, including the performance comparison between dCache [8] and XRootD. Moreover, we present a performance and cost comparison of OpenZFS [9] and Linux software RAID(MD RAID)[10], evaluate different storage software stacks, and provide details on stress tests conducted to validate Third Party Copy (TPC) [11] functionality.…”

Section: Introductionmentioning

confidence: 99%

Exploring Future Storage Options for ATLAS at the BNL/SDCC facility

Huang,

Garonne,

Hancock

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

The ATLAS experiment is expected to deliver an unprecedented amount of scientific data in the High Luminosity(HL-LHC) era. As the demand for disk storage capacity in ATLAS continues to rise steadily, the BNL Scientific Data and Computing Center (SDCC) faces challenges in terms of cost implications for maintaining multiple disk copies and adapting to the coming ATLAS storage requirements. To address these challenges, the SDCC Storage team has undertaken a thorough analysis of the ATLAS experiment’s requirements, matching them to suitable storage options and strategies, and has explored alternatives to enhance or replace the current storage solution. This paper aims to present the main challenges encountered while supporting big data experiments such as ATLAS. We describe the experiment’s specific requirements and priorities, particularly focusing on the critical storage system characteristics of the high-luminosity run and how the key storage components provided by the Storage team work together: the dCache disk storage system; its archival back-end, HPSS; and its OS-level backend Storage. Specifically, we investigate a novel approach to integrate Lustre and XRootD. In this setup, Lustre serves as backend storage and XRootD acts as an access layer frontend, supporting various grid access protocols. Additionally, we also describe the validation and commissioning tests, including the performance comparison between dCache and XRootd. Furthermore, we provide a performance and cost analysis comparing OpenZFS and LINUX MD RAID, evaluate different storage software stacks, and showcase stress tests conducted to validate Third Party Copy (TPC) functionality.

show abstract

Third-party transfers in WLCG using HTTP

Cited by 5 publications

References 15 publications

An HTTP REST API for Tape-backed Storage

An HTTP REST API for Tape-backed Storage

CNAF experience in support of the JUNO distributed computing model

Exploring Future Storage Options for ATLAS at the BNL/SDCC facility

Contact Info

Product

Resources

About