Using TAGs to speed up the ATLAS analysis process

Ehrenfeld, W.; Buckingham, R. M.; Cranshaw, J.; Donszelmann, T. Cuhadar; Doherty, Thomas; Hřivnáć, J.; Nowak, Marcin; Slater, M.; Viegas, F. Tique Aires; Vinek, E.; Zhang, Q

doi:10.1088/1742-6596/331/3/032007

Cited by 8 publications

(6 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The access to an event could be done by the direct access to the event data, or through the index files redirecting to the event data. The advantange of using the index file rather than using the whole set of event data is studied in other experiments, such as the AOD and Mini-AOD in LHC experiments, which allows fast selection and reduces the data size [10,11].…”

Section: Unified Event Data Storementioning

confidence: 99%

Jupyter-based service for JUNO analysis

Lin

2020

EPJ Web Conf.

View full text Add to dashboard Cite

The JUNO (Jiangmen Underground Neutrino Observatory) is designed to determine the neutrino mass hierarchy and precisely measure oscillation parameters. The estimated data volume of raw data is about 2 PB/year. The event rate of reactor anti-neutrinos is about 60/day, while the event rate of background is about O(10) Hz. The challenge is the event correlation during the analysis, where the background events could not be discarded. In order to use big data techniques to search for rare events, a Jupyter-based interactive service is developed for JUNO analysis. In this paper, an overview of this service is presented. The infrastructure is based on Jupyter and Kubernetes, which provides the user interface and resource management. In order to integrate the data processing framework and big data techniques, an index file is used as an intermediate file, which points to the interested events. Data processing framework SNiPER is used to select the candidate of neutrino signals and produce the index file. Apache Spark is then used to process such index file repeatedly with data cached in memory. With the index file produced from Spark and the complete event data files, SNiPER is used to process them and produce the final physics result. At the end of paper, the test-bed is presented and the testing result is shown.

show abstract

Section: Unified Event Data Storementioning

confidence: 99%

Jupyter-based service for JUNO analysis

Lin

2020

EPJ Web Conf.

View full text Add to dashboard Cite

show abstract

“…In addition to ROOT, an interface is also available for writing tuples in HDF5 format [5]. ATLAS also plans to use this mechanism for creating the "tag" database: this is a relational database that holds basic information about each event, to allow rapid selection of events of interest [6].…”

Section: Alternate Storage Formatsmentioning

confidence: 99%

Toolkit for data reduction to tuples for the ATLAS experiment

Snyder

Krasznahorkay

2012

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

The final step in a HEP data-processing chain is usually to reduce the data to a 'tuple' form which can be efficiently read by interactive analysis tools such as ROOT. Often, this is implemented independently by each group analyzing the data, leading to duplicated effort and needless divergence in the format of the reduced data. ATLAS has implemented a common toolkit for performing this processing step. By using tools from this package, physics analysis groups can produce tuples customized for a particular analysis but which are still consistent in format and vocabulary with those produced by other physics groups. The package is designed so that almost all the code is independent of the specific form used to store the tuple. The code that does depend on this is grouped into a set of small backend packages. While the ROOT backend is the most used, backends also exist for HDF5 and for specialized databases. By now, the majority of ATLAS analyses rely on this package, and it is an important contributor to the ability of ATLAS to rapidly analyze physics data.

show abstract

“…Presently, ATLAS sites continue to use HammerCloud to evaluate new configurations, and the functional tests are used to automatically exclude malfunctioning sites from the Panda [4] job brokerage system. Many recent examples of HammerCloud usage can be found in these proceedings [5,6,7,8,9,10,11,12,13].…”

Section: Usage In Lhc Computingmentioning

confidence: 99%

HammerCloud: A Stress Testing System for Distributed Analysis

Ster¹,

Elmsheuser²,

García³

et al. 2011

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Distributed analysis of LHC data is an I/O-intensive activity which places large demands on the internal network, storage, and local disks at remote computing facilities. Commissioning and maintaining a site to provide an efficient distributed analysis service is therefore a challenge which can be aided by tools to help evaluate a variety of infrastructure designs and configurations. HammerCloud is one such tool; it is a stress testing service which is used by central operations teams, regional coordinators, and local site admins to (a) submit arbitrary number of analysis jobs to a number of sites, (b) maintain at a steady-state a predefined number of jobs running at the sites under test, (c) produce web-based reports summarizing the efficiency and performance of the sites under test, and (d) present a web-interface for historical test results to both evaluate progress and compare sites. HammerCloud was built around the distributed analysis framework Ganga, exploiting its API for grid job management. HammerCloud has been employed by the ATLAS experiment for continuous testing of many sites worldwide, and also during large scale computing challenges such as STEP'09 and UAT'09, where the scale of the tests exceeded 10,000 concurrently running and 1,000,000 total jobs over multi-day periods. In addition, HammerCloud is being adopted by the CMS experiment; the plugin structure of HammerCloud allows the execution of CMS jobs using their official tool (CRAB).

show abstract

Using TAGs to speed up the ATLAS analysis process

Cited by 8 publications

References 2 publications

Jupyter-based service for JUNO analysis

Jupyter-based service for JUNO analysis

Toolkit for data reduction to tuples for the ATLAS experiment

HammerCloud: A Stress Testing System for Distributed Analysis

Contact Info

Product

Resources

About