The ATLAS Eventindex: an event catalogue for experiments collecting large amounts of data

Barberis, D.; Cranshaw, J.; Dimitrov, Gancho; Favareto, A.; Casaní, Álvaro Fernández; Hoz, S. González de la; Hřivnáć, J.; Nowak, Marcin; Cairols, José Salt; Sánchez, J.; Sorokoletov, R; Zhang, Q

doi:10.1088/1742-6596/513/4/042002

Cited by 18 publications

(10 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Event Index service [13] of the ATLAS [14] experiment at the Large Hadron Collider (LHC) keeps references to all real and simulated ATLAS events. Hadoop [15] Map files and HBase [16] tables are used to store the Event Index data, a subset of data is also stored in an Oracle database.…”

Section: Atlas Event Indexmentioning

confidence: 99%

Using Graph Databases

Hřivnáć

2020

EPJ Web Conf.

View full text Add to dashboard Cite

Data in High Energy Physics (HEP) usually consist of complext complex data structures stored in relational databases and files with internal schema. Such architecture exhibits many shortcomings, which could be fixed by migrating into Graph Database storage. The paper describes basic principles of the Graph Database together with an overview of existing standards and implementations. The usefulness and usability are demonstrated using the concrete example of the Event Index of the ATLAS experiment at LHC in two approaches as the full storage (all data are in the Graph Database) and meta-storage (a layer of schema-less graph-like data implemented on top of more traditional storage). The usability, the interfaces with the surrounding framework and the performance of those solutions are discussed. The possible more general usefulness for generic experiments’ storage is also discussed.

show abstract

Section: Atlas Event Indexmentioning

confidence: 99%

Using Graph Databases

Hřivnáć

2020

EPJ Web Conf.

View full text Add to dashboard Cite

show abstract

“…Apache Kudu is a next-generation scalable and distributed table-based storage designed for HTAP systems -Hybrid Transactional and Analytical Processing [6]. Unlike most of the data formats available for the Hadoop Distributed File System (HDFS) 1 , Kudu provides indexing and columnar data organization natively -this is to establish a good compromise between random data lookups and analytics performance. The data organization in shared tables with named columns, types and a primary index makes Kudu very attractive for systems with relational data models that need to scale out.…”

Section: Apache Kudumentioning

confidence: 99%

“…The ATLAS EventIndex [1] is a catalogue of all real and simulated events produced by the experiment [2] at all processing stages. The system contains hundreds of billions of event records (180 billion records as of June 2018), each consisting of approximately 1000 bytes.…”

Section: Introductionmentioning

confidence: 99%

A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

et al. 2019

View full text Add to dashboard Cite

The ATLAS EventIndex has been in operation since the beginning of LHC Run 2 in 2015. Like all software projects, its components have been constantly evolving and improving in performance. The main data store in Hadoop, based on MapFiles and HBase, can work for the rest of Run 2 but new solutions are explored for the future. Kudu offers an interesting environment, with a mixture of BigData and relational database features, which look promising at the design level. This environment is used to build a prototype to measure the scaling capabilities as functions of data input rates, total data volumes and data query and retrieval rates. In this proceedings we report on the selected data schemas and on the current performance measurements with the Kudu proto-

show abstract

“…A prototype service has been started in early 2013, in response to the request from the ATLAS 2 experiment, where a new central storage system for all the events produced by the experiment was about to be built [3]. ATLAS wanted to use Hadoop ecosystem as the main data storage and access backend.…”

Section: From a Pilot To Productionmentioning

confidence: 99%

Evolution of the Hadoop Platform and Ecosystem for High Energy Physics

et al. 2019

View full text Add to dashboard Cite

The interest in using scalable data processing solutions based on Apache Hadoop ecosystem is constantly growing in the High Energy Physics (HEP) community. This drives the need for increased reliability and availability of the central Hadoop service and underlying infrastructure provided to the community by the CERN IT department. This paper reports on the overall status of the Hadoop platform and related Hadoop and Spark service at CERN, detailing recent enhancements and features introduced in many areas including the service configuration, availability, alerting, monitoring and data protection, in order to meet the new requirements posed by the users’ community.

show abstract

The ATLAS Eventindex: an event catalogue for experiments collecting large amounts of data

Cited by 18 publications

References 7 publications

Using Graph Databases

Using Graph Databases

A prototype for the evolution of ATLAS EventIndex based on Apache Kudu storage

Evolution of the Hadoop Platform and Ecosystem for High Energy Physics

Contact Info

Product

Resources

About