LCG Persistency Framework (CORAL, COOL, POOL): Status and Outlook

Valassi, A.; Clemencic, M.; Dykstra, Dave; Frank, M.; Front, D; Govi, G.; Kalkhof, A; Loth, A; Nowak, Marcin; Pokorski, Witold; Salnikov, A.; Schmidt, S.; Trentadue, R.; Wache, M.; Xie, Zhijian

doi:10.1088/1742-6596/331/4/042043

Cited by 8 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The detector conditions database contained in the DBRelease package is prepared by copying COOL folders from a central Oracle conditions database to a SQLite file, which is then included in the package. COOL and POOL are components of the LCG Persistency Framework [12]. COOL consists of libraries for handling time variation and versioning of the experiment conditions data, while POOL is a storage solution for complex ROOT [13] objects.…”

Section: Streamlined Environment Setup and Packing Of Conditions And Trigger Datamentioning

confidence: 99%

Containerization in ATLAS Software Development and Data Production

et al. 2021

View full text Add to dashboard Cite

The ATLAS experiment’s software production and distribution on the grid benefits from a semi-automated infrastructure that provides up-to-date information about software usability and availability through the CVMFS distribution service for all relevant systems. The software development process uses a Continuous Integration pipeline involving testing, validation, packaging and installation steps. For opportunistic sites that can not access CVMFS, containerized releases are needed. These standalone containers are currently created manually to support Monte-Carlo data production at such sites. In this paper we will describe an automated procedure for the containerization of ATLAS software releases in the existing software development infrastructure, its motivation, integration and testing in the distributed computing system.

show abstract

Section: Streamlined Environment Setup and Packing Of Conditions And Trigger Datamentioning

confidence: 99%

Containerization in ATLAS Software Development and Data Production

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In addition to the standalone images for simulation we build images for digitization and reconstruction workflows. In this case the conditions database is prepared by copying COOL folders [5] from a central conditions Oracle database to a SQLite file, which is then packaged in the images. This SQLite database file has the minimal payload necessary to execute a reconstruction job, specified by performing a folder dump of the COOL folders that are required to process the input data based on its metadata, such as the run number range and/or the time interval, which specify data taking periods.…”

Section: Atlas Software In Docker Imagesmentioning

confidence: 99%

Standalone containers with ATLAS offline software

et al. 2020

View full text Add to dashboard Cite

This paper describes the deployment of the offline software of the ATLAS experiment at LHC in containers for use in production workflows such as simulation and reconstruction. To achieve this goal we are using Docker and Singularity, which are both lightweight virtualization technologies that can encapsulate software packages inside complete file systems. The deployment of offline releases via containers removes the interdependence between the runtime environment needed for job execution and the configuration of the computing nodes at the sites. Docker or Singularity would provide a uniform runtime environment for the grid, HPCs and for a variety of opportunistic resources. Additionally, releases may be supplemented with a detector’s conditions data, thus removing the need for network connectivity at computing nodes, which is normally quite restricted for HPCs. In preparation to achieve this goal, we have built Docker and Singularity images containing single full releases of ATLAS software for running detector simulation and reconstruction jobs in runtime environments without a network connection. Unlike similar efforts to produce containers by packing all possible dependencies of every possible workflow into heavy images (≈ 200GB), our approach is to include only what is needed for specific workflows and to manage dependencies efficiently via software package managers. This approach leads to more stable packaged releases where the dependencies are clear and the resulting images have more portable sizes ( 16GB). In an effort to cover a wider variety of workflows, we are deploying images that can be used in raw data reconstruction. This is particularly challenging due to the high database resource consumption during the access to the experiment’s conditions payload when processing data. We describe here a prototype pipeline in which images are provisioned only with the conditions payload necessary to satisfy the jobs’ requirements. This database-on-demand approach would keep images slim, portable and capable of supporting various workflows in a standalone fashion in environments with no network connectivity.

show abstract

“…In Run 2 two ATLAS ORACLE Database instances have been used to store the conditions: CONDBR2 [3], keeping conditions for data (online data taking, offline data processing and DCS data), and OFLP200, keeping conditions for Monte Carlo (MC) simulation data production. Every ATLAS sub-system, including TileCal, uses several COOL [4] DB schemas. One of them for TileCal Offline Conditions DB is shown in Figure 2.…”

Section: Tilecal Conditions Database Architecturementioning

confidence: 99%

ATLAS Tile Calorimeter Conditions Database architecture and operations in Run 2

et al. 2020

View full text Add to dashboard Cite

An overview of the Conditions Database (DB) structure for the hadronic Tile Calorimeter (TileCal), one of the sub-systems of the ATLAS detector at LHC, is presented. ATLAS Conditions DB stores the data on the ORACLE backend, and the design and implementation have been developed using the COOL (Conditions Objects for LCG) software package as a common persistency solution for the storage and management of the conditions data. TileCal conditions and calibration data are stored in 4 separate Databases, each with its own schema: TileCal Online and Offline DBs for data, DB for Monte Carlo simulation and Detector Control System (DCS) DB. In order to ensure smooth operation of the TileCal during data taking, experts perform the necessary calibrations, add the changes of detector status and other conditions data, prepare new conditions for data reprocessing and Monte Carlo production campaigns, and upload the new up-to-date information into DB using custom-made software tools. The procedure of TileCal conditions’ preparation, validation, uploading to DBs is described, and some DB-related statistics collected in Run 2 is presented.

show abstract

LCG Persistency Framework (CORAL, COOL, POOL): Status and Outlook

Cited by 8 publications

References 8 publications

Containerization in ATLAS Software Development and Data Production

Containerization in ATLAS Software Development and Data Production

Standalone containers with ATLAS offline software

ATLAS Tile Calorimeter Conditions Database architecture and operations in Run 2

Contact Info

Product

Resources

About