Using the glideinWMS System as a Common Resource Provisioning Layer in CMS

Balcas, Justas; Belforte, S.; Bockelman, Brian; Colling, D.; Gutsche, O.; Hufnagel, D.; Khan, Farrukh Aftab; Larson, Krista; Letts, J.; Mascheroni, Marco; Mason, D.; McCrea, A; Piperov, S.; Saiz-Santos, M; Sfiligoi, I.; Tanasijczuk, A. J.; Wissing, C.

doi:10.1088/1742-6596/664/6/062031

Cited by 25 publications

(23 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…priority of the work, etc. Each WMAgent is responsible for splitting the work into smaller chunks (jobs) and sending them to the CMS Global Pool [4], an HTCondor [5] batch system overlayed, using Glidein-WMS [6], on the CMS-accessible grid resources. The batch system then distributes the jobs to computing resources all over the world-the tiered system mentioned above.…”

Section: Cms Workflow Management Systemmentioning

confidence: 99%

The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

Kuznetsov

Fischer

Guo

2018

Comput Softw Big Sci

View full text Add to dashboard Cite

The CMS experiment at the CERN LHC developed the Workflow Management Archive system to persistently store unstructured framework job report documents produced by distributed workflow management agents. In this paper we present its architecture, implementation, deployment, and integration with the CMS and CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster. The system leverages modern technologies such as a document oriented database and the Hadoop eco-system to provide the necessary flexibility to reliably process, store, and aggregate O(1M) documents on a daily basis. We describe the data transformation, the short and long term storage layers, the query language, along with the aggregation pipeline developed to visualize various performance metrics to assist CMS data operators in assessing the performance of the CMS computing system.

show abstract

Section: Cms Workflow Management Systemmentioning

confidence: 99%

The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

Kuznetsov

Fischer

Guo

2018

Comput Softw Big Sci

View full text Add to dashboard Cite

show abstract

“…CMS Global pool has over 30 schedds connected to it in total, and over a dozen of them are active at the same time. Resource provisioning is based on GlideinWMS [28] which has the knowledge of all the sites supporting CMS computing. DODAS implements a model which de facto bypasses the resource provisioning glidein based and, upon user request, creates a CMS ephemeral WLCG [29] site which auto-registers to the Global Pool.…”

Section: Cms Use Casementioning

confidence: 99%

DODAS: How to effectively exploit heterogeneous clouds for scientific computations

Antonacci¹,

Ciangottini²,

Costantini³

et al. 2018

Proceedings of International Symposium on Grids and Clouds 2018 in Conjunction With Frontiers in Computational Drug Discovery —

View full text Add to dashboard Cite

Dynamic On Demand Analysis Service (DODAS) is a Platform as a Service tool built combining several solutions and products developed by the INDIGO-DataCloud H2020 project. DODAS allows to instantiate on-demand container-based clusters. Both HTCondor batch system and platform for the Big Data analysis based on Spark, Hadoop etc, can be deployed on any cloud-based infrastructures with almost zero effort. DODAS acts as cloud enabler designed for scientists seeking to easily exploit distributed and heterogeneous clouds to process data. Aiming to reduce the learning curve as well as the operational cost of managing community specific services running on distributed cloud, DODAS completely automates the process of provisioning, creating, managing and accessing a pool of heterogeneous computing and storage resources. DODAS was selected as one of the Thematic Services that will provide multidisciplinary solutions in the EOSC-hub project, an integration and management system of the European Open Science Cloud starting in January 2018. The main goals of this contribution are to provide a comprehensive overview of the overall technical implementation of DODAS, as well as to illustrate two distinct real examples of usage: the integration within the CMS Workload Management System and the extension of the AMS computing model.

show abstract

“…As a unified pool [5], it joins geographically distributed resources together under control of a single HTCondor Central Manager (CM) and also provides a single entry point for tasks with diverse resource requests (e.g. single-core and multi-core jobs [6], jobs with higher-thanstandard memory requirements, etc).…”

Section: Introductionmentioning

confidence: 99%

Exploring GlideinWMS and HTCondor scalability frontiers for an expanding CMS Global Pool

et al. 2019

View full text Add to dashboard Cite

The CMS Submission Infrastructure Global Pool, built on Glidein-WMS andHTCondor, is a worldwide distributed dynamic pool responsible for the allocation of resources for all CMS computing workloads. Matching the continuously increasing demand for computing resources by CMS requires the anticipated assessment of its scalability limitations. In addition, the Global Plmust be able to expand in a more heterogeneous environment, in terms of resource provisioning (combining Grid, HPC and Cloud) and workload submissi.A dedicated testbed has been set up to simulate such conditions with the purpose of finding potential bottlenecks in the software or its configuration. This report provides a thorough description of the various scalabilitydimensions in size and complexity that are being explored for the future Global Pool, along with the analysis and solutions to the limitations proposed with the support of the GlideinWMS and HTCondor developer teams.

show abstract

Using the glideinWMS System as a Common Resource Provisioning Layer in CMS

Cited by 25 publications

References 3 publications

The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

DODAS: How to effectively exploit heterogeneous clouds for scientific computations

Exploring GlideinWMS and HTCondor scalability frontiers for an expanding CMS Global Pool

Contact Info

Product

Resources

About