DIRAC universal pilots

Stagni, F.; McNab, A.; Luzzi, C.; Krzemień, W.

doi:10.1088/1742-6596/898/9/092024

Cited by 8 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The DIRAC Workload Management System (WMS) is in charge of exploiting distributed computing resources. In other words, it manages jobs, and pilot jobs [21] (from here on simply called "pilots"). The emergence of new distributed computing resources (private and commercial clouds, High Performance Computing clusters, volunteer computing, etc) changed the traditional landscape of computing for offline processing.…”

Section: The Dirac Projectmentioning

confidence: 99%

Integrating LHCb workflows on HPC resources: status and strategies

Stagni

Valassi²,

Romanovskiy³

2020

EPJ Web Conf.

View full text Add to dashboard Cite

High Performance Computing (HPC) supercomputers are expected to play an increasingly important role in HEP computing in the coming years. While HPC resources are not necessarily the optimal fit for HEP workflows, computing time at HPC centers on an opportunistic basis has already been available to the LHC experiments for some time, and it is also possible that part of the pledged computing resources will be offered as CPU time allocations at HPC centers in the future. The integration of the experiment workflows to make the most efficient use of HPC resources is therefore essential. This paper describes the work that has been necessary to integrate LHCb workflows at a specific HPC site, the Marconi-A2 system at CINECA in Italy, where LHCb benefited from a joint PRACE (Partnership for Advanced Computing in Europe) allocation with the other Large Hadron Collider (LHC) experiments. This has required addressing two types of challenges: on the software application workloads, for optimising their performance on a many-core hardware architecture that differs significantly from those traditionally used in WLCG (Worldwide LHC Computing Grid), by reducing memory footprint using a multi-process approach; and in the distributed computing area, for submitting these workloads using more than one logical processor per job, which had never been done yet in LHCb.

show abstract

Section: The Dirac Projectmentioning

confidence: 99%

Integrating LHCb workflows on HPC resources: status and strategies

Stagni

Valassi²,

Romanovskiy³

2020

EPJ Web Conf.

View full text Add to dashboard Cite

show abstract

“…The DIRAC Workload Management System (WMS) is in charge of exploiting distributed computing resources. In other words, it manages jobs, and pilot jobs [4] (from here on simply called "pilots").…”

Section: Exploiting Computing Resourcesmentioning

confidence: 99%

The DIRAC interware: current, upcoming and planned capabilities and technologies

et al. 2020

View full text Add to dashboard Cite

Efficient access to distributed computing and storage resources is mandatory for the success of current and future High Energy and Nuclear Physics Experiments. DIRAC is an interware to build and operate distributed computing systems. It provides a development framework and a rich set of services for the Workload, Data and Production Management tasks of large scientific communities. A single DIRAC installation provides a complete solution for the distributed computing of one, or more than one collaboration. The DIRAC Workload Management System (WMS) provides a transparent, uniform interface for managing computing resources. The DIRAC Data Management System (DMS) offers all the necessary tools to ensure data handling operations: it supports transparent access to storage resources based on multiple technologies, and is easily expandable. Distributed Data management can be performed, also using third party services, and operations are resilient with respect to failures. DIRAC is highly customizable and can be easily extended. For these reasons, a vast and heterogeneous set of scientific collaborations have adopted DIRAC as the base for their computing models. Users from different experiments can interact with the system in different ways, depending on their specific tasks, expertise level and previous experience using command line tools, python APIs or Web Portals. The requirements of the diverse DIRAC user communities and hosting infrastructures triggered multiple developments to improve the system usability: examples include the adoption of industry standard authorization and authentication infrastructure solutions, the management of diverse computing resources (cloud, HPC, GPGPU, etc.), the handling of high-intensity work and data flows, but also advanced monitoring and accounting using no-SQL based solutions and message queues. This contribution will highlight DIRAC’s current, upcoming and planned capabilities and technologies.

show abstract

“…The DIRAC system is installed on worker nodes (WN) by distributed agents called pilots [11,12]. The development of a universal and scalable logging system for all pilots is also foreseen to accommodate the use of the MQ (see Fig.…”

Section: Message Queues In Diracmentioning

confidence: 99%

Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware

et al. 2019

View full text Add to dashboard Cite

The Message Queue (MQ) architecture is an asynchronous communication scheme that provides an attractive solution for certain scenarios in a distributed computing model. The introduction of MQ as an intermediate component in-between the interacting processes allows to decouple the end-points making the system more flexible and providing high scalability and redundancy. DIRAC is a general-purpose interware software for distributed computing systems, which offers a common interface to a number of heterogeneous providers and guarantees transparent and reliable usage of the resources. The DIRAC platform has been adapted by several scientific projects, including High Energy Physics communities like LHCb, the Linear Collider and Belle2. A Message Queue generic interface has been incorporated into the DIRAC framework to help solving the scalability challenges that must be addressed during LHC Run3, starting in 2021. It allows to use the MQ scheme for a message exchange among the DIRAC components or to communicate with third-party services. Within this contribution we describe the integration of MQ systems with DIRAC and several use cases are shown. Message Queues are foreseen to be used in the pilot logging system, and as a backbone of the DIRAC component logging system and monitoring. *

show abstract

DIRAC universal pilots

Cited by 8 publications

References 9 publications

Integrating LHCb workflows on HPC resources: status and strategies

Integrating LHCb workflows on HPC resources: status and strategies

The DIRAC interware: current, upcoming and planned capabilities and technologies

Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware

Contact Info

Product

Resources

About