An Oracle-based event index for ATLAS

Dimitrov, Gancho; Vasileva, Petya; Baranowski, Zbigniew; Canali, Luca; Dumitru, Adrian; Formica, A.

doi:10.1088/1742-6596/898/4/042033

Cited by 6 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the performance in retrieving information from the Hadoop system was still suboptimal at the end of 2015, it was decided to add an alternative store, based on an Oracle database, to support a well-defined set of queries [8]. All real events now have their metadata also stored in an Oracle database, albeit without trigger information.…”

Section: System Architecturementioning

confidence: 99%

The ATLAS EventIndex for LHC Run 3

et al. 2020

View full text Add to dashboard Cite

The ATLAS EventIndex was designed in 2012-2013 to provide a global event catalogue and limited event-level metadata for ATLAS analysis groups and users during the LHC Run 2 (2015-2018). It provides a good and reliable service for the initial use cases (mainly event picking) and several additional ones, such as production consistency checks, duplicate event detection and measurements of the overlaps of trigger chains and derivation datasets. The LHC Run 3, starting in 2021, will see increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. This proceeding describes the implementation of a new core storage service that will be able to provide at least the same functionality as the current one for increased data ingestion and search rates, and with increasing volumes of stored data. It is based on a set of HBase tables, with schemas derived from the current Oracle implementation, coupled to Apache Phoenix for data access; in this way we will add to the advantages of a BigData based storage system the possibility of SQL as well as NoSQL data access, allowing to re-use most of the existing code for metadata integration.

show abstract

Section: System Architecturementioning

confidence: 99%

The ATLAS EventIndex for LHC Run 3

et al. 2020

View full text Add to dashboard Cite

show abstract

“…However, the use cases evolved in the meantime and have been extended from event picking and production completeness checks to trigger overlap studies, duplicate event detection and derivation streams (offline triggers) overlaps. At the same time, the implementation of fast data querying based on a traditional relational database involves only a subset of information and is available for real events [4]. Also the event rate increased steadily throughout Run 2.…”

Section: Motivation For the Project Evolutionmentioning

confidence: 99%

A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

et al. 2019

View full text Add to dashboard Cite

The ATLAS EventIndex has been in operation since the beginning of LHC Run 2 in 2015. Like all software projects, its components have been constantly evolving and improving in performance. The main data store in Hadoop, based on MapFiles and HBase, can work for the rest of Run 2 but new solutions are explored for the future. Kudu offers an interesting environment, with a mixture of BigData and relational database features, which look promising at the design level. This environment is used to build a prototype to measure the scaling capabilities as functions of data input rates, total data volumes and data query and retrieval rates. In this proceedings we report on the selected data schemas and on the current performance measurements with the Kudu proto-

show abstract

“…The ATLAS Collaboration adopted HBase to build event index database EventIndex [4]. Additionally, the ATLAS collaboration used Oracle relational database to build EventIndex Oracle-based system (EIO) [5] for providing event-wise services. The ATLAS EventIndex uses Hadoop technologies for efficiently storing all indexed data, and a subset is copied into Oracle for accessing.…”

Section: Related Workmentioning

confidence: 99%

EventDB: an event-based indexer and caching system for BESIII experiment

Cheng

et al. 2019

EPJ Web Conf.

View full text Add to dashboard Cite

Beijing Spectrometer (BESIII) experiment has produced hundreds of billions of events. The traditional event-wise accessing of BESIII Offline Software System is not effective for the selective accessing with low rate during a physics analysis. In this paper, an event-based data management system (EventDB) is introduced, which can effectively alleviate the problems of low efficiency of data processing and low utilization of resources. Firstly, an indexing system based on NoSQL database is designed. By extracting specified attributes of events, the events interested to the physicists are selected and stored into the database, whilst the real data of event is still stored in ROOT files. For those hot events, the real event data can also be cached into EventDB to improve the access performance. The data analysis workflow of HEP experiments is needed to change if the EventDB system is applied. The analysis program queries the corresponding event index from database, then get event data from database if the event is cached, or get data from ROOT files if it is not cached. Finally, the test on more than one hundred billion physics events shows the query speed was greatly improved over traditional file-based data management systems.

show abstract

An Oracle-based event index for ATLAS

Cited by 6 publications

References 8 publications

The ATLAS EventIndex for LHC Run 3

The ATLAS EventIndex for LHC Run 3

A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

EventDB: an event-based indexer and caching system for BESIII experiment

Contact Info

Product

Resources

About