Improving the Scheduling Efficiency of a Global Multi-Core HTCondor Pool in CMS

Bockelman, Brian; Foyo, Diego Davila; Anampa, K. Hurtado; Ivanov, T.; Khan, Farrukh Aftab; Kotobi, Amjad; Larson, Krista; Letts, J.; Mascheroni, Marco; Mason, D.; Yzquierdo, A. Pérez-Calero

doi:10.1051/epjconf/201921403056

Cited by 3 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Global Pool matches diverse CMS workloads, which include single-core as well as multicore requests [378] to these resources. A successful scheduling is achieved by simultaneously ensuring that all available resources are efficiently used [379], a fair share of resources between users is reached, and the completion of CMS tasks follows their prioritization, minimizing job failures and manual intervention. While the SI typically manages 100k to 150k simultaneously executing tasks, recent scalability tests [380] have demonstrated the capacity of the infrastructure to sustain in excess of half a million concurrently running jobs.…”

Section: Central Processing and Productionmentioning

confidence: 99%

Development of the CMS detector for the CERN LHC Run 3

Hayrapetyan,

Tumasyan,

Adam

et al. 2024

J. Inst.

View full text Add to dashboard Cite

Since the initial data taking of the CERN LHC, the CMS experiment has undergone substantial upgrades and improvements. This paper discusses the CMS detector as it is configured for the third data-taking period of the CERN LHC, Run 3, which started in 2022. The entire silicon pixel tracking detector was replaced. A new powering system for the superconducting solenoid was installed. The electronics of the hadron calorimeter was upgraded. All the muon electronic systems were upgraded, and new muon detector stations were added, including a gas electron multiplier detector. The precision proton spectrometer was upgraded. The dedicated luminosity detectors and the beam loss monitor were refurbished. Substantial improvements to the trigger, data acquisition, software, and computing systems were also implemented, including a new hybrid CPU/GPU farm for the high-level trigger.

show abstract

Section: Central Processing and Productionmentioning

confidence: 99%

Development of the CMS detector for the CERN LHC Run 3

Hayrapetyan,

Tumasyan,

Adam

et al. 2024

J. Inst.

View full text Add to dashboard Cite

show abstract

“…Similarly, one can find examples of workload abstraction for HTCondor in HEP as well. Collaborations such as CMS and ATLAS use higher-level concepts such as pilot jobs to steer the work of batch pools across their sites world-wide [96,97]. Sometimes these tools go even further and provide programmatic interfaces to help users in organising all the files needed for an analysis, as is the case with the ATLAS PanDA and the LHCb GANGA systems [97,98].…”

Section: State Of the Artmentioning

confidence: 99%

Distributed Computing Solutions for High Energy Physics Interactive Data Analysis

Padulano¹

View full text Add to dashboard Cite

The scientific research in High Energy Physics (HEP) is characterised by complex computational challenges, which over the decades had to be addressed by researching computing techniques in parallel to the advances in understanding physics. One of the main actors in the field, CERN, hosts both the Large Hadron Collider (LHC) and thousands of researchers yearly who are devoted to collecting and processing the huge amounts of data generated by the particle accelerator. This has historically provided a fertile ground for distributed computing techniques, which led to the creation of the Worldwide LHC Computing Grid (WLCG), a global network providing large computing power for all the experiments revolving around the LHC and the HEP field. Data generated by the LHC so far has already posed challenges for computing and storage. vii de recerca sobre aquesta proposta. Des del punt de vista de l'usuari, això es detalla en forma duna nova interfície que es pot executar en un ordinador portàtil o en milers de nodes informàtics, sense canvis en l'aplicació de l'usuari. Aquest desenvolupament obre la porta a l'explotació de recursos distribuïts a través de motors d'execució estàndard de la indústria que poden escalar a múltiples nodes en clústers HPC o HTC, o fins i tot en ofertes serverless de núvols comercials. Atès que sovint l'anàlisi de dades en aquest camp està limitada per E/S, cal comprendre quins són els possibles mecanismes d'emmagatzematge en memòria cau. En aquest sentit, es va investigar un nou sistema d'emmagatzematge basat en la tecnologia d'emmagatzematge d'objectes com a objectiu per a la memòria cau.En conclusió, el futur de l'anàlisi de dades a HEP presenta reptes des de diverses perspectives, des de l'explotació de recursos informàtics i d'emmagatzematge distribuïts fins al disseny d'interfícies d'usuari ergonòmiques. Els marcs de programari han d'apuntar a l'eficiència i la facilitat d'ús, desvinculant la definició dels càlculs físics dels detalls d'implementació de la seva execució. Aquesta tesi s'emmarca en l'esforç col•lectiu de la comunitat HEP cap a aquests objectius, definint problemes i possibles solucions que poden ser adoptades per futurs investigadors. viii

show abstract

Exploring GlideinWMS and HTCondor scalability frontiers for an expanding CMS Global Pool

et al. 2019

View full text Add to dashboard Cite

The CMS Submission Infrastructure Global Pool, built on Glidein-WMS andHTCondor, is a worldwide distributed dynamic pool responsible for the allocation of resources for all CMS computing workloads. Matching the continuously increasing demand for computing resources by CMS requires the anticipated assessment of its scalability limitations. In addition, the Global Plmust be able to expand in a more heterogeneous environment, in terms of resource provisioning (combining Grid, HPC and Cloud) and workload submissi.A dedicated testbed has been set up to simulate such conditions with the purpose of finding potential bottlenecks in the software or its configuration. This report provides a thorough description of the various scalabilitydimensions in size and complexity that are being explored for the future Global Pool, along with the analysis and solutions to the limitations proposed with the support of the GlideinWMS and HTCondor developer teams.

show abstract

Improving the Scheduling Efficiency of a Global Multi-Core HTCondor Pool in CMS

Cited by 3 publications

References 6 publications

Development of the CMS detector for the CERN LHC Run 3

Development of the CMS detector for the CERN LHC Run 3

Distributed Computing Solutions for High Energy Physics Interactive Data Analysis

Exploring GlideinWMS and HTCondor scalability frontiers for an expanding CMS Global Pool

Contact Info

Product

Resources

About