MONIT: Monitoring the CERN Data Centres and the WLCG Infrastructure

The University of California system maintains excellent networking between its campuses and a number of other Universities in California, including Caltech, most of them being connected at 100 Gbps. UCSD and Caltech Tier2 centers have joined their disk systems into a single logical caching system, with worker nodes from both sites accessing data from disks at either site. This successful setup has been in place for the last two years. However, coherently managing nodes at multiple physical locations is not trivial and requires an update on the operations model used. The Pacific Research Platform (PRP) provides Kubernetes resource pool spanning resources in the science demilitarized zones (DMZs) in several campuses in California and worldwide. We show how we migrated the XCache services from bare-metal deployments into containers using the PRP cluster. This paper presents the reasoning behind our hardware decisions and the experience in migrating to and operating in a mixed environment.

show abstract

“…1. The job monitoring data from the global pool that can be accessed through MONIT [6] (i.e. which dataset was accessed by each job).…”

Section: Monitoringmentioning

confidence: 99%

“…Monthly global working set size for analysis jobs using MINIAOD and MINIAODSIM data tiers. Data obtained from MONIT[6].…”

mentioning

confidence: 99%

Moving the California distributed CMS XCache from bare metal into containers using Kubernetes

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The monitoring infrastructure at CERN [7], known as MONIT, is based on the following architecture (see also Fig. 1):…”

Section: The Monitoring Infrastructure At Cernmentioning

confidence: 99%

Big data solutions for CMS computing monitoring and analytics

2020

View full text Add to dashboard Cite

The CMS computing infrastructure is composed of several subsystems that accomplish complex tasks such as workload and data management, transfers, submission of user and centrally managed production requests. Till recently, most subsystems were monitored through custom tools and web applications, and logging information was scattered over several sources and typically accessible only by experts. In the last year, CMS computing fostered the adoption of common big data solutions based on open-source, scalable, and no-SQL tools, such as Hadoop, InfluxDB, and ElasticSearch, available through the CERN IT infrastructure. Such systems allow for the easy deployment of monitoring and accounting applications using visualisation tools such as Kibana and Grafana. Alarms can be raised when anomalous conditions in the monitoring data are met, and the relevant teams are automatically notified. Data sources from different subsystems are used to build complex workflows and predictive analytics (such as data popularity, smart caching, transfer latency), and for performance studies. We describe the full software architecture and data flow, the CMS computing data sources and monitoring applications, and show how the stored data can be used to gain insights into the various subsystems by exploiting scalable solutions based on Spark.

show abstract

“…Cooperation between experiments within WLCG is encouraged and the sharing of monitoring infrastructure has been a subject for this cooperation. An example of this shared infrastructure is the MONIT deployment at CERN [2]. This system relies heavily on time-series histograms to display the changing value of a large number of metrics.…”

Section: Beyond Time-series Histogramsmentioning

confidence: 99%

Monitoring distributed computing beyond the traditional time-series histogram

2020

View full text Add to dashboard Cite

In this work we describe a novel approach to monitor the operation of distributed computing services. Current monitoring tools are dominated by the use of time-series histograms showing the evolution of various metrics. These can quickly overwhelm or confuse the viewer due to the large number of similar looking graphs. We propose a supplementary approach through the sonification of real-time data streamed directly from a variety of distributed computing services. The real-time nature of this method allows operations staff to quickly detect problems and identify that a problem is still ongoing, avoiding the case of investigating an issue a-priori when it may already have been resolved. In this paper we present details of the system architecture and provide a recipe for deployment suitable for both site and experiment teams.

show abstract

MONIT: Monitoring the CERN Data Centres and the WLCG Infrastructure

Cited by 21 publications

References 5 publications

Moving the California distributed CMS XCache from bare metal into containers using Kubernetes

Moving the California distributed CMS XCache from bare metal into containers using Kubernetes

Big data solutions for CMS computing monitoring and analytics

Monitoring distributed computing beyond the traditional time-series histogram

Contact Info

Product

Resources

About