The Data Quality Monitoring Software for the CMS experiment at the LHC: past, present and future

Abstract

The Data Quality Monitoring software is a central tool in the CMS experiment. It is used in the following key environments: (i) Online, for real-time detector monitoring; (ii) Offline, for the prompt-offline-feedback and final fine-grained data quality analysis and certification; (iii) Validation of all the reconstruction software production releases; (iv) Validation in Monte Carlo productions. Though the basic structure of the Run1 DQM system remains the same for Run2, between the Run1 and Run2 periods, the DQM system underwent substantial upgrades in many areas, not only to adapt to the surrounding infrastructure changes, but also to provide improvements to meet the growing needs of the collaboration with an emphasis on more sophisticated methods for evaluating data quality. We need to cope with the higher-energy and -luminosity proton-proton collision data, as well as the data from various special runs, such as Heavy Ion runs. In this contribution, we will describe the current DQM software, structure and workflow in the different environments. We then discuss the performance and our experiences with the DQM system in Run2. The main technical challenges which we have encountered and the solutions adopted during Run2 will also be discussed, including efficient use of memory in multithreading environments. Finally, we present the prospect of a future DQM upgrade with emphasis on functionality and long-term robustness for LHC Run3.

Machine Learning applications for Data Quality Monitoring and Data Certification within CMS

“…To maintain a good amount of quality data for physics analyses, CMS Collaboration has DQM [5] and DC workflows. Both workflows oversee live DQM during data-taking (also called online operations), and data certification after the data has been recorded from the detector and fully reconstructed (also called offline operations During collisions, the CMS detector sends a small portion of the data stream to the online DQM backend.…”

HL-LHC Analysis With ROOT: ROOT Project Input to the HL-LHC Computing Review (Stage 2)

“…Nonetheless, without sustained effort for two years, a new ROOT histogram library cannot be advanced to the minimal feature level required for adoption. The community risks continued work around the ownership discrepancies of ROOT's histograms, paying a performance price for analyses, and additional fragmentation in the experiments' online and offline framework, where even today, experiment specific histogramming facilities have been reintroduced to not use ROOT's histogram library [25]. CMS has provided a review of issues of ROOT histograms for DQM.…”

Evaluating Mixed-Precision Arithmetic for 3D Generative Adversarial Networks to Simulate High Energy Physics Detectors

“…In the recent years, several studies have demonstrated the benefit of using Deep Learning (DL) to solve typical tasks related to data taking and analysis. Building on these examples, many HEP experiments are now working on integrating DL into their workflows for many different applications: from data quality assurance [38], to real-time selection of interesting collision events [39], to simulation [40] and data analysis [41]. For example, generative models, from GAN to Variational Auto-Encoders (VAE), are being tested as fast alternatives to Monte Carlo based simulation.…”