A Roadmap for HEP Software and Computing R&amp;D for the 2020s

Elsen, E.

doi:10.1007/s41781-019-0031-6

Cited by 30 publications

(20 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The updated accelerator, scheduled to begin taking data in 2027, is planned to collect around 30 times more data than the LHC has produced so far. As this amount is close to an exabyte, it is clear that future data management will require approaches beyond simply scaling current solutions, given the future budget estimations and the expected technological evolution [2]. This means that software will have to bridge the gap between technology evolution and real computing needs, making the most out of current and future architectures.…”

Section: Introductionmentioning

confidence: 99%

Distributed data analysis with ROOT RDataFrame

et al. 2020

View full text Add to dashboard Cite

Widespread distributed processing of big datasets has been around for more than a decade now thanks to Hadoop, but only recently higher-level abstractions have been proposed for programmers to easily operate on those datasets, e.g. Spark. ROOT has joined that trend with its RDataFrame tool for declarative analysis, which currently supports local multi-threaded parallelisation. However, RDataFrame’s programming model is general enough to accommodate multiple implementations or backends: users could write their code once and execute it as-is locally or distributedly, just by selecting the corresponding backend. This abstract introduces PyRDF, a new python library developed on top of RDataFrame to seamlessly switch from local to distributed environments with no changes in the application code. In addition, PyRDF has been integrated with a service for web-based analysis, SWAN, where users can dynamically plug in new resources, as well as write, execute, monitor and debug distributed applications via an intuitive interface.

show abstract

Section: Introductionmentioning

confidence: 99%

Distributed data analysis with ROOT RDataFrame

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Data analysis in experimental particle physics involves studying the result of particle collisions in detectors and comparing experimental results with theoretical models. The computational data processing can be roughly categorised in several stages (Albrecht et al, 2019) illustrated in Figure 1. In the data-taking stage, the data are filtered by selecting events of interest using a multi-tiered trigger system reconstructing physics objects with increasing precision.…”

Section: Introductionmentioning

confidence: 99%

Scalable Declarative HEP Analysis Workflows for Containerised Compute Clouds

Šimko¹,

Heinrich²,

Lange³

et al. 2021

Front. Big Data

View full text Add to dashboard Cite

We describe a novel approach for experimental High-Energy Physics (HEP) data analyses that is centred around the declarative rather than imperative paradigm when describing analysis computational tasks. The analysis process can be structured in the form of a Directed Acyclic Graph (DAG), where each graph vertex represents a unit of computation with its inputs and outputs, and the graph edges describe the interconnection of various computational steps. We have developed REANA, a platform for reproducible data analyses, that supports several such DAG workflow specifications. The REANA platform parses the analysis workflow and dispatches its computational steps to various supported computing backends (Kubernetes, HTCondor, Slurm). The focus on declarative rather than imperative programming enables researchers to concentrate on the problem domain at hand without having to think about implementation details such as scalable job orchestration. The declarative programming approach is further exemplified by a multi-level job cascading paradigm that was implemented in the Yadage workflow specification language. We present two recent LHC particle physics analyses, ATLAS searches for dark matter and CMS jet energy correction pipelines, where the declarative approach was successfully applied. We argue that the declarative approach to data analyses, combined with recent advancements in container technology, facilitates the portability of computational data analyses to various compute backends, enhancing the reproducibility and the knowledge preservation behind particle physics data analyses.

show abstract

“…It is foreseen that it will generate roughly thirty times more data than the LHC has produced so far. Given the available future budget estimations and expected technological evolutions [3], experiments at the LHC and the larger HEP community would benefit from improvements on the software. It will be crucial to make the most out of current and future architectures.…”

Section: Introductionmentioning

confidence: 99%

Fine-grained data caching approaches to speedup a distributed RDataFrame analysis

2021

View full text Add to dashboard Cite

Thanks to its RDataFrame interface, ROOT now supports the execution of the same physics analysis code both on a single machine and on a cluster of distributed resources. In the latter scenario, it is common to read the input ROOT datasets over the network from remote storage systems, which often increases the time it takes for physicists to obtain their results. Storing the remote files much closer to where the computations will run can bring latency and execution time down. Such a solution can be improved further by caching only the actual portion of the dataset that will be processed on each machine in the cluster, reusing it in subsequent executions on the same input data. This paper shows the benefits of applying different means of caching input data in a distributed ROOT RDataFrame analysis. Two such mechanisms will be applied to this kind of workflow with different configurations, namely caching on the same nodes that process data or caching on a separate server.

show abstract

A Roadmap for HEP Software and Computing R&D for the 2020s

Cited by 30 publications

References 1 publication

Distributed data analysis with ROOT RDataFrame

Distributed data analysis with ROOT RDataFrame

Scalable Declarative HEP Analysis Workflows for Containerised Compute Clouds

Fine-grained data caching approaches to speedup a distributed RDataFrame analysis

Contact Info

Product

Resources

About