Overview of the ATLAS distributed computing system

Elmsheuser, J.; Girolamo, A. Di

doi:10.1051/epjconf/201921403010

Cited by 9 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The HCDC model targets the bandwidth and access latency bottlenecks of the data carousel model. The HCDC model can be used by the continuous ATLAS derivation production workflow [1]. In this model only tape storage would contain permanent replicas of the input data.…”

Section: Simulation Of Cloud Data Managementmentioning

confidence: 99%

“…The distributed computing system [1] of the ATLAS experiment [2] at the LHC is built around two main components: the workflow management system PanDA [3] and the data management system Rucio [4]. The involved systems manage the computing resources to process the detector data at the Tier-0 at CERN, reprocesses it periodically at the distributed Tier-1 and Tier-2 Worldwide LHC Computing Grid (WLCG) [5] sites, and runs continuous Monte Carlo (MC) simulation and reconstruction.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Seamless integration of commercial Clouds with ATLAS Distributed Computing

Megino

Harinder

et al. 2021

EPJ Web Conf.

Self Cite

View full text Add to dashboard Cite

The CERN ATLAS Experiment successfully uses a worldwide distributed computing Grid infrastructure to support its physics programme at the Large Hadron Collider (LHC). The Grid workflow system PanDA routinely manages up to 700,000 concurrently running production and analysis jobs to process simulation and detector data. In total more than 500 PB of data are distributed over more than 150 sites in the WLCG and handled by the ATLAS data management system Rucio. To prepare for the ever growing data rate in future LHC runs new developments are underway to embrace industry accepted protocols and technologies, and utilize opportunistic resources in a standard way. This paper reviews how the Google and Amazon Cloud computing services have been seamlessly integrated as a Grid site within PanDA and Rucio. Performance and brief cost evaluations will be discussed. Such setups could offer advanced Cloud tool-sets and provide added value for analysis facilities that are under discussions for LHC Run-4.

show abstract

Section: Simulation Of Cloud Data Managementmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Seamless integration of commercial Clouds with ATLAS Distributed Computing

Megino

Harinder

et al. 2021

EPJ Web Conf.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Figure 1: At left: the WLCG infrastructure which, across its Tier0, Tier1, and Tier2 centers, provided the bulk of the computing capacity for LHC Run 1 and Run 2 processing, storage, simulation and analysis. At right, distributed computing system components (in this case ATLAS [3]) which are typical of the LHC experiments.…”

Section: Scale Of Operationsmentioning

confidence: 99%

Towards a NoOps Model for WLCG

et al. 2020

View full text Add to dashboard Cite

One of the most costly factors in providing a global computing infrastructure such as the WLCG is the human effort in deployment, integration, and operation of the distributed services supporting collaborative computing, data sharing and delivery, and analysis of extreme scale datasets. Furthermore, the time required to roll out global software updates, introduce new service components, or prototype novel systems requiring coordinated deployments across multiple facilities is often increased by communication latencies, staff availability, and in many cases expertise required for operations of bespoke services. While the WLCG (and distributed systems implemented throughout HEP) is a global service platform, it lacks the capability and flexibility of a modern platform-as-a-service including continuous integration/continuous delivery (CI/CD) methods, development-operations capabilities (DevOps, where developers assume a more direct role in the actual production infrastructure), and automation. Most importantly, tooling which reduces required training, bespoke service expertise, and the operational effort throughout the infrastructure, most notably at the resource endpoints (sites), is entirely absent in the current model. In this paper, we explore ideas and questions around potential NoOps models in this context: what is realistic given organizational policies and constraints? How should operational responsibility be organized across teams and facilities? What are the technical gaps? What are the social and cybersecurity challenges? Conversely what advantages does a NoOps model deliver for innovation and for accelerating the pace of delivery of new services needed for the HL-LHC era? We will describe initial work along these lines in the context of providing a data delivery network supporting IRIS-HEP DOMA R&D.

show abstract

“…The distributed computing system of the ATLAS experiment [1] at the LHC is built around the two main components: the workflow management system PanDA and the data manage- ment system Rucio [2]. It manages the computing resources to process the detector data at the Tier-0 at CERN, reprocesses it once per year at the Tier-1 and Tier-2 Worldwide LHC Computing Grid (WLCG) [3] sites and runs continuous Monte Carlo (MC) simulation and reconstruction.…”

Section: Introductionmentioning

confidence: 99%

Evolution of the ATLAS analysis model for Run-3 and prospects for HL-LHC

et al. 2020

Self Cite

View full text Add to dashboard Cite

With an increased dataset obtained during the Run-2 of the LHC at CERN, the even larger forthcoming Run-3 data and the expected increase of the dataset by more than one order of magnitude for the HL-LHC, the ATLAS experiment is reaching the limits of the current data production model in terms of disk storage resources. The anticipated availability of an improved fast simulation will enable ATLAS to produce significantly larger Monte Carlo samples with the available CPU, which will then be limited by insuﬃcient disk resources. The ATLAS Analysis Model Study Group for Run-3 was setup at the end of Run-2. Its tasks have been to analyse the eﬃciency and suitability of the current analysis model and to propose significant improvements. The group has considered options allowing ATLAS to save, for the same sample of data and simulated events, at least 30% disk space overall, and has given recommendations on how significantly larger savings could be realised for the HL-LHC. Furthermore, suggestions were made to harmonise the current stage of analysis across the collaboration. The group has now completed its work: key recommendations will be the new small sized analysis formats DAOD_PHYS and DAOD_PHYSLITE and the increased usage of a tape carousel mode in the centralised production of these formats. This proceeding reviews the recommended ATLAS analysis model for Run-3 and the status of its implementation. It also provides an outlook to the HL-LHC analysis.

show abstract

Overview of the ATLAS distributed computing system

Cited by 9 publications

References 5 publications

Seamless integration of commercial Clouds with ATLAS Distributed Computing

Seamless integration of commercial Clouds with ATLAS Distributed Computing

Towards a NoOps Model for WLCG

Evolution of the ATLAS analysis model for Run-3 and prospects for HL-LHC

Contact Info

Product

Resources

About