gLExec: gluing grid computing to the Unix world

Abstract. After a scheduled maintenance and upgrade period, the world's largest and most powerful machine -the Large Hadron Collider(LHC) -is about to enter its second run at unprecedented energies. In order to exploit the scientific potential of the machine, the experiments at the LHC face computational challenges with enormous data volumes that need to be analysed by thousand of physics users and compared to simulated data. Given diverse funding constraints, the computational resources for the LHC have been deployed in a worldwide mesh of data centres, connected to each other through Grid technologies. The PanDA (Production and Distributed Analysis) system was developed in 2005 for the ATLAS experiment on top of this heterogeneous infrastructure to seamlessly integrate the computational resources and give the users the feeling of a unique system. Since its origins, PanDA has evolved together with upcoming computing paradigms in and outside HEP, such as changes in the networking model, Cloud Computing and HPC. It is currently running steadily up to 200 thousand simultaneous cores (limited by the available resources for ATLAS), up to two million aggregated jobs per day and processes over an exabyte of data per year. The success of PanDA in ATLAS is triggering the widespread adoption and testing by other experiments. In this contribution we will give an overview of the PanDA components and focus on the new features and upcoming challenges that are relevant to the next decade of distributed computing workload management using PanDA.

show abstract

“…Tests to run the job with gLExec [6] identity switching have been completed successfully, but it is not mandatory;…”

Section: Overviewmentioning

confidence: 99%

PanDA: Exascale Federation of Resources for the ATLAS Experiment at the LHC

Megino

Bejar

et al. 2016

EPJ Web of Conferences

View full text Add to dashboard Cite

show abstract

“…In the absence of solutions based on the virtualization of resources (VM), CREAM implements isolation via local credential mapping, exploiting traditional Unix-level security mechanisms like a separate user account per Grid user or per job. This Unix domain isolation is implemented in the form of the gLExec system [24], a sudo-style program which allows the execution of the user's job with local credentials derived from the user's identity and any accompanying authorization assertions. This relation between the Grid credentials and the local Unix accounts and groups is determined by the Local Credential MAPping Service (LCMAPS) [25].…”

Section: Securitymentioning

confidence: 99%

Design and implementation of the gLite CREAM job management service

Aiftimiei

Andreetto

Bertocco

et al. 2010

Future Generation Computer Systems

View full text Add to dashboard Cite

Job execution and management is one of the most important functionality provided by every modern Grid systems. In this paper we describe how the problem of job management has been addressed in the gLite middleware by means of the CREAM and CEMonitor services. CREAM (Computing Resource Execution and Management) provides a job execution and management capability for Grids, while CEMonitor is a general-purpose asynchronous event notification framework. Both components expose a Web Service interface allowing conforming clients to submit, manage and monitor computational jobs to a Local Resource Management System.

show abstract

“…Site managers have expressed concerns about potential security problems caused by the changing of the ownership of the workload executed on a Worker Node. However, the introduction of a glexec authorization tool which can be used on the Worker Nodes to apply site policies to the user workload obtained by the generic pilot agent resolves this problem [14]. …”

Section: Managing Vo Policiesmentioning

confidence: 99%

DIRAC: a community grid solution

Tsaregorodtsev

Bargiotti

Brook

et al. 2008

J. Phys.: Conf. Ser.

123

View full text Add to dashboard Cite

Abstract. The DIRAC system was developed in order to provide a complete solution for using the distributed computing resources of the LHCb experiment at CERN for data production and analysis. It allows a concurrent use of over 10K CPUs and 10M file replicas distributed over many tens of sites. The sites can be part of a Computing Grid such as WLCG or standalone computing clusters all integrated in a single management structure. DIRAC is a generic system with the LHCb specific functionality incorporated through a number of plug-in modules. It can be easily adapted to the needs of other communities. Special attention is paid to the resilience of the DIRAC components to allow an efficient use of non-reliable resources. The DIRAC production management components provide a framework for building highly automated data production systems including data distribution and data driven workload scheduling. In this paper we give an overview of the DIRAC system architecture and design choices. We show how different components are put together to compose an integrated data processing system including all the aspects of the LHCb experiment -from the MC production and raw data reconstruction to the final user analysis.

show abstract

gLExec: gluing grid computing to the Unix world

Cited by 27 publications

References 4 publications

PanDA: Exascale Federation of Resources for the ATLAS Experiment at the LHC

PanDA: Exascale Federation of Resources for the ATLAS Experiment at the LHC

Design and implementation of the gLite CREAM job management service

DIRAC: a community grid solution

Contact Info

Product

Resources

About