Harvester : an edge service harvesting heterogeneous resources for ATLAS

Maeno, T.; Megino, Fernando Harald Barreiro; Benjamin, D.; Cameron, D.; Childers, J. T.; De, K.; Salvo, A. De; Filipčić, A.; Hover, John; Lin, FaHui; Oleynik, D.

doi:10.1051/epjconf/201921403030

Cited by 17 publications

(16 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The stand-alone Raythena version presented in this work needs to be extended so that it can be automatically launched once workload is available. This will be achieved by connecting it with the Harvester application [6] which retrieves task information from the Panda server. Further, the Pilot2 [7] application will be used to handle AthenaMP process instead of the stand-alone solution presented in this work.…”

Section: Ray-based Atlas Event Service Applicationmentioning

confidence: 99%

Raythena: a vertically integrated scheduler for ATLAS applications on heterogeneous distributed resources

et al. 2020

View full text Add to dashboard Cite

The ATLAS experiment has successfully integrated HighPerformance Computing resources (HPCs) in its production system. Unlike the current generation of HPC systems, and the LHC computing grid, the next generation of supercomputers is expected to be extremely heterogeneous in nature: different systems will have radically different architectures, and most of them will provide partitions optimized for different kinds of workloads. In this work we explore the applicability of concepts and tools realized in Ray (the high-performance distributed execution framework targeting large-scale machine learning applications) to ATLAS event throughput optimization on heterogeneous distributed resources, ranging from traditional grid clusters to Exascale computers. We present a prototype of Raythena, a Ray-based implementation of the ATLAS Event Service (AES), a fine-grained event processing workflow aimed at improving the efficiency of ATLAS workflows on opportunistic resources, specifically HPCs. The AES is implemented as an event processing task farm that distributes packets of events to several worker processes running on multiple nodes. Each worker in the task farm runs an event-processing application (Athena) as a daemon. The whole system is orchestrated by Ray, which assigns work in a distributed, possibly heterogeneous, environment. For all its flexibility, the AES implementation is currently comprised of multiple separate layers that communicate through ad-hoc command-line and filebased interfaces. The goal of Raythena is to integrate these layers through a feature-rich, efficient application framework. Besides increasing usability and robustness, a vertically integrated scheduler will enable us to explore advanced concepts such as dynamically shaping of workflows to exploit currently available resources, particularly on heterogeneous systems.

show abstract

Section: Ray-based Atlas Event Service Applicationmentioning

confidence: 99%

Raythena: a vertically integrated scheduler for ATLAS applications on heterogeneous distributed resources

et al. 2020

View full text Add to dashboard Cite

show abstract

“…To exploit all available resources an efficient job scheduling is necessary. The rather diverse resource usage capability of the ATLAS workflow management will be improved with developments of Harvester [12], Event Service and Event Streaming Service [13] and a new version of the PanDA pilot as described further in this section.…”

Section: Evolution Of the Systemsmentioning

confidence: 99%

Overview of the ATLAS distributed computing system

Elmsheuser

Girolamo

2019

EPJ Web Conf.

View full text Add to dashboard Cite

The CERN ATLAS experiment successfully uses a worldwide computing infrastructure to support the physics program during LHC Run 2. The Grid workflow system PanDA routinely manages 250 to 500 thousand concurrently running production and analysis jobs to process simulation and detector data. In total more than 370 PB of data is distributed over more than 150 sites in the WLCG and handled by the ATLAS data management system Rucio. To prepare for the ever growing LHC luminosity in future runs new developments are underway to even more efficiently use opportunistic resources such as HPCs and utilize new technologies. This paper will review and explain the outline and the performance of the ATLAS distributed computing system and give an outlook to new workflow and data management ideas for the beginning of the LHC Run 3. It will be discussed that the ATLAS workflow and data management systems are robust, performant and can easily cope with the higher Run 2 LHC performance. There are presently no scaling issues and each subsystem is able to sustain the large loads. * Panda Rucio Grid CPU HPCs CPU Clouds CPU ProdSys User AGIS Workflows Jobs Configuration Data Monitoring, Analytics EPJ Web of Conferences 214, 03010 (2019) https://doi.org/10.1051/epjconf/201921403010 CHEP 2018been added. In addition the activities within the WLCG DOMA (data organisation, management, access) project [14] to explore possibilities to overcome disk shortage in HL-LHC are ramping up and ATLAS is actively participating. A very successful R&D project with Google has been started which allows the integration of Rucio and distributed analysis using Harvester [15].The site infrastructure is evolving as well and the support for containers using Singularity [16] and Docker [17] is under development at the sites and within PanDA jobs for production and user analysis usage.

show abstract

“…• Push mode: works naturally on Harvester [6] or ARC Control Tower [7]. This mode loses late binding.…”

Section: Unified Panda Queuesmentioning

confidence: 99%

ATLAS Global Shares implementation in PanDA

Megino

Girolamo

et al. 2019

EPJ Web Conf.

Self Cite

View full text Add to dashboard Cite

PanDA (Production and Distributed Analysis) is the workload management system for ATLAS across the Worldwide LHC Computing Grid. While analysis tasks are submitted to PanDA by over a thousand users following personal schedules (e.g. PhD or conference deadlines), production campaigns are scheduled by a central Physics Coordination group based on the organization's calendar. The Physics Coordination group needs to allocate the amount of Grid resources dedicated to each activity, in order to manage sharing of CPU resources among various parallel campaigns and to make sure that results can be achieved in time for important deadlines. While dynamic and static shares on batch systems have been around for a long time, we are trying to move away from local resource partitioning and manage shares at a global level in the PanDA system. The global solution is not straightforward, given different requirements of the activities (number of cores, memory, I/O and CPU intensity), the heterogeneity of Grid resources (site/HW capabilities, batch configuration and queue setup) and constraints on data locality. We have therefore started the Global Shares project that follows a requirements-driven multi-step execution plan, starting from definition of nestable shares, implementing share-aware job dispatch, aligning internal processes with global shares and finally implementing a pilot stream control for controlling the batch slots that keeps late binding. This contribution will explain the development work and architectural changes in PanDA to implement Global Shares, and describe how the Global Shares project has enabled the central control of resources and significantly reduced manual operations.

show abstract

Harvester : an edge service harvesting heterogeneous resources for ATLAS

Cited by 17 publications

References 7 publications

Raythena: a vertically integrated scheduler for ATLAS applications on heterogeneous distributed resources

Raythena: a vertically integrated scheduler for ATLAS applications on heterogeneous distributed resources

Overview of the ATLAS distributed computing system

ATLAS Global Shares implementation in PanDA

Contact Info

Product

Resources

About