Managing the ATLAS Grid through Harvester

Megino, Fernando Harald Barreiro; Alekseev, A. A.; Berghaus, F.; Cameron, D.; De, K.; Filipčić, A.; Glushkov, I.; Lin, FaHui; Maeno, T.; Magini, N.

doi:10.1051/epjconf/202024503010

Cited by 5 publications

(4 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We assume that for each experiment an HTCondor server is available and that the experiments' computing systems are already configured to send jobs to a specific HTCondor system, for example ATLAS submits via Harvester [12] and Belle-II via a DIRAC site director [13].…”

Section: Csvas a Wlcg Compute Sitementioning

confidence: 99%

Managing remote cloud resources for multiple HEP VOs with cloudscheduler

Driemel,

Ebert,

Sobie

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

Cloudscheduler is a system to manage resources of local and remote compute clouds and makes those resources available to HTCondor pools. It examines the resource needs of idle jobs, then starts virtual machines (VMs), sized accordingly, on allowed clouds with available resources. Using yaml files, cloudscheduler then provisions the VMs during the boot process with all necessary tools needed to register with HTCondor and run the experiment’s jobs. Although we have run cloudscheduler in its first version for ATLAS and Belle-II workloads successfully for more than 10 years, we developed cloudscheduler version 2 (CSV2), a complete overhaul and modernization of cloudscheduler. The new system is used successfully in production for Belle-II, ATLAS, DUNE, and BABAR. In addition to using cloudscheduler version 2 as a WLCG site, we also run it as a service for other WLCG sites, and the Canadian Advanced Network for Astronomical Research (CANFAR) group uses its own instance of CSV2 for their astronomy workloads. In this paper, we report on our experience in operating CSV2 for different experiment’s jobs, running on up to 10,000 cores across all experiments and clouds in North America, Australia, and Europe. We will also report on how to correctly account for the resource usage in the WLCG APEL system, how the monitoring works, as well as on the integration of different clouds and how to use resources opportunistically.

show abstract

Section: Csvas a Wlcg Compute Sitementioning

confidence: 99%

Managing remote cloud resources for multiple HEP VOs with cloudscheduler

Driemel,

Ebert,

Sobie

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

show abstract

“…To mitigate this problem, it was decided to use Elasticsearch, Logstash and Kibana (ELK) stack for global monitoring of Harvester. The data is copied every 10 minutes from the PanDA DB to the central Elasticsearch storage provided by CERN IT [11]. The implemented views of the Harvester monitoring module are mostly used for debugging immediate problems related to a particular worker or PanDA job because BigPanDA monitor reads data directly from the PanDA database without additional delays.…”

Section: Harvester Monitoring Modulementioning

confidence: 99%

BigPanDA monitoring system evolution in the ATLAS Experiment

Korchuganova,

Alekseev,

Klimentov

et al. 2024

EPJ Web of Conf.

Self Cite

View full text Add to dashboard Cite

Monitoring services play a crucial role in the day-to-day operation of distributed computing systems. The ATLAS Experiment at LHC uses the Production and Distributed Analysis workload management system (PanDA WMS), which allows a million computational jobs to run daily at over 170 computing centers of the WLCG and opportunistic resources, utilizing 600k cores simultaneously on average. The BigPanDA monitor is an essential part of the monitoring infrastructure for the ATLAS Experiment that provides a wide range of views, from top-level summaries to a single computational job and its logs. Over the past few years of the PanDA WMS advancement in the ATLAS Experiment, several new components were developed, such as Harvester, iDDS, Data Carousel, and Global Shares. Due to its modular architecture, the BigPanDA monitor naturally grew into a platform where the relevant data from all PanDA WMS components and accompanying services are accumulated and displayed in the form of interactive charts and tables. Moreover the system has been adopted by other experiments beyond HEP. In this paper we describe the evolution of the BigPanDA monitor system, the development of new modules, and the integration process into other experiments.

show abstract

“…The workload management system PanDA is used to process, broker and dispatch all ATLAS workloads across the heterogeneous WLCG and opportunistic resources. The interface to the resources is handled by Harvester [14], which was implemented following a flexible plugin approach so that it can be easily expanded for any available resource. During the first proof of concept phase with Google in 2017 [15] the ATLAS payloads on Google Compute Engine (GCE) had to be integrated in the most native and lightweight way possible.…”

Section: Integration Of Kubernetes With the Atlas Workload Management Systemmentioning

confidence: 99%

Seamless integration of commercial Clouds with ATLAS Distributed Computing

Megino

Harinder

et al. 2021

EPJ Web Conf.

Self Cite

View full text Add to dashboard Cite

The CERN ATLAS Experiment successfully uses a worldwide distributed computing Grid infrastructure to support its physics programme at the Large Hadron Collider (LHC). The Grid workflow system PanDA routinely manages up to 700,000 concurrently running production and analysis jobs to process simulation and detector data. In total more than 500 PB of data are distributed over more than 150 sites in the WLCG and handled by the ATLAS data management system Rucio. To prepare for the ever growing data rate in future LHC runs new developments are underway to embrace industry accepted protocols and technologies, and utilize opportunistic resources in a standard way. This paper reviews how the Google and Amazon Cloud computing services have been seamlessly integrated as a Grid site within PanDA and Rucio. Performance and brief cost evaluations will be discussed. Such setups could offer advanced Cloud tool-sets and provide added value for analysis facilities that are under discussions for LHC Run-4.

show abstract

Managing the ATLAS Grid through Harvester

Cited by 5 publications

References 5 publications

Managing remote cloud resources for multiple HEP VOs with cloudscheduler

Managing remote cloud resources for multiple HEP VOs with cloudscheduler

BigPanDA monitoring system evolution in the ATLAS Experiment

Seamless integration of commercial Clouds with ATLAS Distributed Computing

Contact Info

Product

Resources

About