Smart Caching at CMS: applying AI to XCache edge services

Abstract

The projected Storage and Compute needs for the HL-LHC will be a factor up to 10 above what can be achieved by the evolution of current technology within a flat budget. The WLCG community is studying possible technical solutions to evolve the current computing in order to cope with the requirements; one of the main focus is resource optimization, with the ultimate aim of improving performance and efficiency, as well as simplifying and reducing operation costs. As of today the storage consolidation based on a Data Lake model is considered a good candidate for addressing HL-LHC data access challenges. The Data Lake model under evaluation can be seen as a logical system that hosts a distributed working set of analysis data. Compute power can be “close” to the lake, but also remote and thus completely external. In this context we expect data caching to play a central role as a technical solution to reduce the impact of latency and reduce network load. A geographically distributed caching layer will be functional to many satellite computing centers that might appear and disappear dynamically. In this talk we propose a system of caches, distributed at national level, describing both deployment and results of the studies made to measure the impact on the CPU efficiency. In this contribution, we also present the early results on novel caching strategy beyond the standard XRootD approach whose results will be a baseline for an AI-based smart caching system.

Reinforcement Learning for Smart Caching at the CMS experiment

“…Once that the advantages related to caches are evident in the CMS experiment context, the main goal is to enhance the efficiency of the cache system with a smart caching approach. Thus, the subject of Smart Caching project [12] is the development of such a smart cache system using AI algorithms within the CMS experiment. The main idea is to develop an AI system that directly manipulates the cache file content, deciding what to write or delete.…”

“…CMS already extensively tested the benefits deriving from the use of a cache system [12]. In particular it has been shown that cache systems allow to reduce the overall network traffic, thus making the processing jobs more efficient by reducing I/O wait time for remote data.…”

New storage and data access solution for CMS experiment in Spain towards HL-LHC era

“…Conclusions and outlook Studies on the popularity of CMS data read by jobs that are executed in the Spanish region have been presented, with a quantitative identification of the most suitable files for being stored in cache systems, based on the repeatability of accesses. Further tasks can include Machine Learning techniques to analyze with more precision the most popular datasets or predict their popularity beforehand [12]. Based on the historical information of accesses, the cache service has been first simulated, in order to find the optimal configuration to keep popular files at both PIC and CIEMAT sites.…”