FTS improvements for LHC Run-3 and beyond

Karavakis, E.; Manzi, Andrea; Rios, Maria Arsuaga; Keeble, Oliver; Cabot, Carles Garcia; Simon, M.; Patrascoiu, Mihai; Angelogiannopoulos, Aris

doi:10.1051/epjconf/202024504016

Cited by 21 publications

(21 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rucio also manages the orchestration of transfers to enable coordinated physics processing as well as user analysis and long-term archival. The actual data movements are executed by the File Transfer Service (FTS) [7], which is one of the transfer tools with which Rucio communicates.…”

Section: Distributed Data Managementmentioning

confidence: 99%

“…• Phase II: Workflow management and data management systems integration, and integration of the ATLAS specific distributed software components with middleware, such as the File Transfer Service (FTS) [7], and with Grid sites. During Phase II we identified missing distributed software components needed for an e↵ective Data Carousel operation (it was mostly related to Rucio/ProdSys2 integration and a special service to release data processing before the whole data sample is staged in).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The ATLAS Data Carousel Project Status

et al. 2021

Self Cite

View full text Add to dashboard Cite

The High Luminosity upgrade to the LHC, which aims for a tenfold increase in the luminosity of proton-proton collisions at an energy of 14 TeV, is expected to start operation in 2028/29 and will deliver an unprecedented volume of scientific data at the multi-exabyte scale. This amount of data has to be stored, and the corresponding storage system must ensure fast and reliable data delivery for processing by scientific groups distributed all over the world. The present LHC computing and data management model will not be able to provide the required infrastructure growth, even taking into account the expected hardware technology evolution. To address this challenge, the Data Carousel R&D project was launched by the ATLAS experiment in the fall of 2018. State-of-the-art data and workflow management technologies are under active development, and their current status is presented here.

show abstract

Section: Distributed Data Managementmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The ATLAS Data Carousel Project Status

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The File Transfer Service (FTS3) [1] distributes the majority of the Large Hadron Collider (LHC) [2] data across the Worldwide LHC Computing Grid (WLCG) [3] infrastructure. It is integrated with experiment frameworks such as Rucio [4] and DIRAC [5] and it is used by more than 35 experiments at CERN and in other data-intensive sciences outside of the LHC and even outside the High Energy Physics (HEP) domain.…”

Section: Fts3mentioning

confidence: 99%

FTS3: Data Movement Service in containers deployed in OKD

et al. 2021

Self Cite

View full text Add to dashboard Cite

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider’s data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the configuration currently running at Fermilab.

show abstract

“…The CERN IT Storage group (IT-ST) provides many services to the physics community. In this introduction, we will describe the three key components used by the four LHC experiments: the EOS disk storage system [1], its archival back-end, the CERN Tape Archive (CTA) [2], and the File Transfer Service (FTS) [3], which are essential for LHC data taking, data processing, data distribution, and analysis.…”

Section: Introductionmentioning

confidence: 99%

LHC Data Storage: Preparing for the Challenges of Run-3

et al. 2021

Self Cite

View full text Add to dashboard Cite

The CERN IT Storage Group ensures the symbiotic development and operations of storage and data transfer services for all CERN physics data, in particular the data generated by the four LHC experiments (ALICE, ATLAS, CMS and LHCb). In order to accomplish the objectives of the next run of the LHC (Run-3), the Storage Group has undertaken a thorough analysis of the experiments’ requirements, matching them to the appropriate storage and data transfer solutions, and undergoing a rigorous programme of testing to identify and solve any issues before the start of Run-3. In this paper, we present the main challenges presented by each of the four LHC experiments. We describe their workflows, in particular how they communicate with and use the key components provided by the Storage Group: the EOS disk storage system; its archival back-end, the CERN Tape Archive (CTA); and the File Transfer Service (FTS). We also describe the validation and commissioning tests that have been undertaken and challenges overcome: the ATLAS stress tests to push their DAQ system to its limits; the CMS migration from PhEDEx to Rucio, followed by large-scale tests between EOS and CTA with the new FTS “archive monitoring” feature; the LHCb Tier-0 to Tier-1 staging tests and XRootD Third Party Copy (TPC) validation; and the erasure coding performance in ALICE.

show abstract

FTS improvements for LHC Run-3 and beyond

Cited by 21 publications

References 15 publications

The ATLAS Data Carousel Project Status

The ATLAS Data Carousel Project Status

FTS3: Data Movement Service in containers deployed in OKD

LHC Data Storage: Preparing for the Challenges of Run-3

Contact Info

Product

Resources

About