Using Continous Deployment techniques to manage software change at a WLCG Tier-2

Abstract

Continuous Integration (CI) and Continuous Development (CD) are common techniques in software development. Continuous Integration is the practice of bringing together code from multiple developers into a single repository, while Continuous Development is the process by which new releases are automatically created and tested. CI/CD pipelines are available in popular automation tools such as GitLab, and act to enhance and accelerate the software development process. Continuous Deployment, in which automation is employed to push new software releases into the production environment, follows naturally from CI/CD, but is not as well established due to business and legal requirements. Such requirements do not exist in the Worldwide LHC Compute Gird (WLCG), making the use of continuous deployment to simplify the management of grid resources an attractive proposition. We have developed work presented previously on containerised worker node environments by introducing continuous deployment techniques and tooling, and show how these, in conjunction with CI/CD, can reduce the management burden at a WLCG Tier-2 resource. In particular, benefits include reduced downtime as a result of code changes and middleware updates.

Versatile software-defined HPC and cloud clusters on Alps supercomputer for diverse workflows

“…Grid middleware communities have been gradually adopting cloud-native technologies for the development and deployment across WLCG data centres. For instance, (Roy et al, 2020) demonstrated the management burden is reduced and uptime has increased by adopting cloud-native technologies such as containerisation and container orchestration systems. This work has similarities to our approach for adopting standard CI/CD tools.…”

“…Previous studies have demonstrated that the middleware for virtualised computing and storage stacks aren’t interoperable with classic, Petascale supercomputing systems (De et al, 2021) (Boccali et al 2021). The middleware has been partially containerised (Roy et al, 2020). Our vCluster approach maps the entire middleware using three compute and storage management solutions, namely, Slurm, Kuberntes and Ceph (on-premise software-defined storage for HTC-friendly data services) (Ceph).…”