Raythena: a vertically integrated scheduler for ATLAS applications on heterogeneous distributed resources

Abstract

The ATLAS experiment has successfully integrated HighPerformance Computing resources (HPCs) in its production system. Unlike the current generation of HPC systems, and the LHC computing grid, the next generation of supercomputers is expected to be extremely heterogeneous in nature: different systems will have radically different architectures, and most of them will provide partitions optimized for different kinds of workloads. In this work we explore the applicability of concepts and tools realized in Ray (the high-performance distributed execution framework targeting large-scale machine learning applications) to ATLAS event throughput optimization on heterogeneous distributed resources, ranging from traditional grid clusters to Exascale computers. We present a prototype of Raythena, a Ray-based implementation of the ATLAS Event Service (AES), a fine-grained event processing workflow aimed at improving the efficiency of ATLAS workflows on opportunistic resources, specifically HPCs. The AES is implemented as an event processing task farm that distributes packets of events to several worker processes running on multiple nodes. Each worker in the task farm runs an event-processing application (Athena) as a daemon. The whole system is orchestrated by Ray, which assigns work in a distributed, possibly heterogeneous, environment. For all its flexibility, the AES implementation is currently comprised of multiple separate layers that communicate through ad-hoc command-line and filebased interfaces. The goal of Raythena is to integrate these layers through a feature-rich, efficient application framework. Besides increasing usability and robustness, a vertically integrated scheduler will enable us to explore advanced concepts such as dynamically shaping of workflows to exploit currently available resources, particularly on heterogeneous systems.

Evolution of HEP Processing Frameworks

“…A further challenge is to keep track of the progress of many independent applications processing small chunks of events, where the applications have the ability to stop work after any chunk has finished or use some other mechanism to deal with early termination. One way to achieve this goal is to integrate the application framework with a workflow management system as demonstrated by the Raythena/EventService project [14].…”

Towards a distributed heterogeneous task scheduler for the ATLAS offline software framework

“…As an intermediate step, we have implemented Raythena [5] a task farm managing multiple AthenaMP processes within a single HPC allocation. This layer of scheduling requires a lot of boilerplate components to stay compatible with the WLCG.…”