Transparent Integration of Opportunistic Resources into the WLCG Compute Infrastructure

Böhler, Michael; Caspart, R.; Fischer, Max; Freyermuth, O.; Giffels, M.; Kroboth, Stefan; Kuehn, Eileen; Schnepf, M.; Cube, Florian von; Wienemann, P.

doi:10.1051/epjconf/202125102039

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To provide seamless, transparent and uniform access to these variety of community supplied heterogeneous compute resources for diverse communities, the Compute4PUNCH concept has been developed, following the idea of establishing a nationwide federated heterogeneous compute infrastructure for the PUNCH4NFDI sciences, adopting already existing technologies developed within the context of federated computing infrastructures in the German High Energy Physics (HEP) community [2] as much as possible in order to maximize synergies.…”

Section: Federated Heterogeneous Compute Infrastructure -Compute4punchmentioning

confidence: 99%

Federated Heterogeneous Compute and Storage Infrastructure for the PUNCH4NFDI Consortium

Drabent,

Freyermuth,

Giffels

et al. 2024

EPJ Web of Conf.

Self Cite

View full text Add to dashboard Cite

PUNCH4NFDI, funded by the Germany Research Foundation initially for five years, is a diverse consortium of particle, astro-, astroparticle, hadron and nuclear physics embedded in the National Research Data Infrastructure initiative. In order to provide seamless and federated access to the huge variety of compute and storage systems provided by the participating communities covering their very diverse needs, the Compute4PUNCH and Storage4PUNCH concepts have been developed. Both concepts comprise state-of-the-art technologies such as a token-based AAI for standardized access to compute and storage resources. The community supplied heterogeneous HPC, HTC and Cloud compute resources are dynamically and transparently integrated into one federated HTCondorbased overlay batch system using the COBalD/TARDIS resource meta-scheduler. Traditional login nodes and a JupyterHub provide entry points into the entire landscape of available compute resources, while container technologies and the CERN Virtual Machine File System (CVMFS) ensure a scalable provisioning of community-specific software environments. In Storage4PUNCH, community supplied storage systems mainly based on dCache or XRootD technology are being federated in a common infrastructure employing methods that are well established in the wider HEP community. Furthermore existing technologies for caching as well as metadata handling are being evaluated with the aim for a deeper integration. The combined Compute4PUNCH and Storage4PUNCH environment will allow a large variety of researchers to carry out resource-demanding analysis tasks. In this contribution we will present the Compute4PUNCH and Storage4PUNCH concepts, the current status of the developments as well as first experiences with scientific applications being executed on the available prototypes.

show abstract

Section: Federated Heterogeneous Compute Infrastructure -Compute4punchmentioning

confidence: 99%

Federated Heterogeneous Compute and Storage Infrastructure for the PUNCH4NFDI Consortium

Drabent,

Freyermuth,

Giffels

et al. 2024

EPJ Web of Conf.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Otherwise, no new drones are launched and the resources are returned to the UBS. Figure 1 shows an example of two clusters connected via COBalD/TARDIS [3]. Here, the UBS is an HPC cluster that provides additional resources for the OBS, the HTC cluster.…”

Section: Introductionmentioning

confidence: 99%

AUDITOR: Accounting for opportunistic resources

Boehler,

Gamel,

Kroboth

et al. 2024

EPJ Web of Conf.

Self Cite

View full text Add to dashboard Cite

The increasing computational demand in High Energy Physics (HEP) as well as increasing concerns about energy efficiency in highperformance/high-throughput computing are driving forces in the search for more efficient ways to utilise available resources. Since avoiding idle resources is key in achieving high efficiency, an appropriate measure is sharing of idle resources of underutilised sites with fully occupied sites. The software COBalD/TARDIS can automatically, transparently, and dynamically (dis)integrate such resources in an opportunistic manner. Sharing resources however also requires accounting. In this work we introduce AUDITOR (AccoUnting DatahandlIng Toolbox for Opportunistic Resources), a flexible and extensible accounting system that is able to cover a wide range of use cases and infrastructures. AUDITOR gathers accounting data via so-called collectors which are designed to monitor batch systems, COBalD/TARDIS, cloud schedulers, or other sources of information. The data is stored in a database and provided to so-called plugins, which act based on accounting records. An action could for instance be creating a bill of utilised resources, computing the CO2 footprint, adjusting parameters of a service, or forwarding accounting information to other accounting systems. Depending on the use case, a suitable collector and plugin can be chosen from a growing ecosystem of collectors and plugins. Libraries for interacting with AUDITOR are provided to facilitate the development of collectors and plugins by the community.

show abstract

Transparent Integration of Opportunistic Resources into the WLCG Compute Infrastructure

Cited by 2 publications

References 9 publications

Federated Heterogeneous Compute and Storage Infrastructure for the PUNCH4NFDI Consortium

Federated Heterogeneous Compute and Storage Infrastructure for the PUNCH4NFDI Consortium

AUDITOR: Accounting for opportunistic resources

Contact Info

Product

Resources

About