Improving efficiency of analysis jobs in CMS

Ivanov, T.; Belforte, S.; Wolf, M.; Mascheroni, Marco; Yzquierdo, A. Pérez-Calero; Letts, J.; Hernández, J. M.; Cristella, L.; Ciangottini, D.; Balcas, Justas; Woodard, A.; Anampa, K. Hurtado; Bockelman, Brian; Foyo, Diego Davila

doi:10.1051/epjconf/201921403006

Cited by 3 publications

(3 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In studies we found that 99% of centralized production jobs finish within their estimated wall clock time, while analysis jobs, which are subject to much more uncertainty due to user code contributions, were found to finish within three hours of their estimated time with 99% confidence. This 3 hour uncertainty in analysis job run time estimation came after many improvements during the year [7]. These improvements allowed the SI group to shorten the retirement time to 4h from 10h, improving scheduling efficiency by several percent, and allowing >99% of jobs to complete within the pilot lifetime.…”

Section: Pilot Improvementsmentioning

confidence: 99%

Improving the Scheduling Efficiency of a Global Multi-Core HTCondor Pool in CMS

et al. 2019

Self Cite

View full text Add to dashboard Cite

Scheduling multi-core workflows in a global HTCondor pool is a multi-dimensional problem whose solution depends on the requirements of the job payloads, the characteristics of available resources, and the boundary conditions such as fair share and prioritization imposed on the job matching to resources. Within the context of a dedicated task force, CMS has increased significantly the scheduling efficiency of workflows in reusable multi-core pilots by various improvements to the limitations of the GlideinWMS pilots, accuracy of resource requests, efficiency and speed of the HTCondor infrastructure, and job matching algorithms.

show abstract

Section: Pilot Improvementsmentioning

confidence: 99%

Improving the Scheduling Efficiency of a Global Multi-Core HTCondor Pool in CMS

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…The workload management system executes payloads in compute nodes provisioned through GlideinWMS [3] and made available as execution slots in a Vanilla Universe HTCondor pool [4]. HTCondor jobs are submitted via specific workload management tools: WMAgent for central data processing and Monte Carlo production jobs, and CRAB for user jobs [5]. The data management system is modular and includes several components: PhEDEx, the data transfer and location system; DBS, the Data Bookkeeping Service, a metadata catalog; and DAS, the Data Aggregation Service designed to aggregate views and provide them to users and services [6].…”

Section: Introductionmentioning

confidence: 99%

The CMS monitoring infrastructure and applications

Ariza-Porras

Kuznetsov

Legger

2021

Comput Softw Big Sci

View full text Add to dashboard Cite

The globally distributed computing infrastructure required to cope with the multi-petabyte datasets produced by the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) at CERN comprises several subsystems, such as workload management, data management, data transfers, and submission of users’ and centrally managed production requests. To guarantee the efficient operation of the whole infrastructure, CMS monitors all subsystems according to their performance and status. Moreover, we track key metrics to evaluate and study the system performance over time. The CMS monitoring architecture allows both real-time and historical monitoring of a variety of data sources. It relies on scalable and open source solutions tailored to satisfy the experiment’s monitoring needs. We present the monitoring data flow and software architecture for the CMS distributed computing applications. We discuss the challenges, components, current achievements, and future developments of the CMS monitoring infrastructure.

show abstract

“…The workload management system executes payloads in compute nodes provisioned through GlideinWMS [3] and thus made available as execution slots in a Vanilla Universe HTCondor [4]. HTCondor jobs are submitted via specific workload management tools: WMAgent for central data processing and Monte Carlo production jobs, and CRAB for user jobs [5]. The data management system is modular.…”

Section: Introductionmentioning

confidence: 99%

Big data solutions for CMS computing monitoring and analytics

2020

View full text Add to dashboard Cite

The CMS computing infrastructure is composed of several subsystems that accomplish complex tasks such as workload and data management, transfers, submission of user and centrally managed production requests. Till recently, most subsystems were monitored through custom tools and web applications, and logging information was scattered over several sources and typically accessible only by experts. In the last year, CMS computing fostered the adoption of common big data solutions based on open-source, scalable, and no-SQL tools, such as Hadoop, InfluxDB, and ElasticSearch, available through the CERN IT infrastructure. Such systems allow for the easy deployment of monitoring and accounting applications using visualisation tools such as Kibana and Grafana. Alarms can be raised when anomalous conditions in the monitoring data are met, and the relevant teams are automatically notified. Data sources from different subsystems are used to build complex workflows and predictive analytics (such as data popularity, smart caching, transfer latency), and for performance studies. We describe the full software architecture and data flow, the CMS computing data sources and monitoring applications, and show how the stored data can be used to gain insights into the various subsystems by exploiting scalable solutions based on Spark.

show abstract

Improving efficiency of analysis jobs in CMS

Cited by 3 publications

References 8 publications

Improving the Scheduling Efficiency of a Global Multi-Core HTCondor Pool in CMS

Improving the Scheduling Efficiency of a Global Multi-Core HTCondor Pool in CMS

The CMS monitoring infrastructure and applications

Big data solutions for CMS computing monitoring and analytics

Contact Info

Product

Resources

About