Evaluating query languages and systems for high-energy physics data

Graur, Dan; Müller, Ingo; Fourny, Ghislain; Watts, G.; Proffitt, M. L.; Alonso, Gustavo

doi:10.14778/3489496.3489498

Cited by 10 publications

(28 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternative data formats continue to be used in HEP, although it is unclear whether their adoption is increasing. As analyses are probably the most agile part of HEP's software environment, they are expected to continue to try alternatives (HDF5, Parquet, etc), also in function of the tools and libraries used by analyses [6]. ROOT's goal is to preempt such re-formatting, which has consistently proven as a bottleneck for analyses' agility, preventing smooth integration of optimizations in the ROOT-part of the analyses, and increasing the storage needs for analyses.…”

Section: Data Formatmentioning

confidence: 99%

HL-LHC Analysis With ROOT: ROOT Project Input to the HL-LHC Computing Review (Stage 2)

Naumann

Canal

Tejedor

et al. 2022

View full text Add to dashboard Cite

Section: Data Formatmentioning

confidence: 99%

HL-LHC Analysis With ROOT: ROOT Project Input to the HL-LHC Computing Review (Stage 2)

Naumann

Canal

Tejedor

et al. 2022

View full text Add to dashboard Cite

“…Although many tools exist to address data analysis needs of industries and academia, it should not be taken for granted that they can all work just as well in any other field. Particularly, it has been shown that for the HEP data analysis requirements a tailor-made tool like ROOT with its RDataFrame data analysis interface still has a major advantage over other industry frameworks [24]. The HEP field is not new to the investigation of large-scale distributed execution engines.…”

Section: Related Workmentioning

confidence: 99%

Leveraging state-of-the-art engines for large-scale data analysis in High Energy Physics

Padulano

Kabadzhov

Saavedra

et al. 2022

Preprint

View full text Add to dashboard Cite

The Large Hadron Collider (LHC) at CERN has generated a vast amount of information from physics events, reaching peaks of TB of data per day which are then sent to large storage facilities. Traditionally, data processing workflows in the High Energy Physics (HEP) field have leveraged grid computing resources. In this context, users have been responsible for manually parallelising the analysis, sending tasks to computing nodes and aggregating the partial results. Analysis environments in this field have had a common building block in the ROOT software framework. This is the de facto standard tool for storing, processing and visualising HEP data. ROOT offers a modern analysis tool called RDataFrame, which can parallelise computations from a single machine to a distributed cluster while hiding most of the scheduling and result aggregation complexity from users. This is currently done by leveraging Apache Spark as the distributed execution engine, but other alternatives are being explored by HEP research groups. Notably, Dask has rapidly gained popularity thanks to its ability to interface with batch queuing systems, widespread in HEP grid computing facilities. Furthermore, future upgrades of the LHC are expected to bring a dramatic increase in data volumes. This paper presents a novel implementation of the Dask backend for the distributed RDataFrame tool in order to address the aforementioned future trends. The scalability of the tool with both the new backend and the already available Spark backend is demonstrated for the first time on more than two thousand cores, testing a real HEP analysis.

show abstract

“…In other systems, this requires different tools [33] [12], because trying to do all of this in the same program is very cumbersome due to the many impedance mismatches. A benchmark [17] was also performed to show the limitations and shortcomings of other query languages and APIs, with a focus on nestedness and a use case in high-energy physics.…”

Section: Data Preparation With Jsoniqmentioning

confidence: 99%

RumbleML: program the lakehouse with JSONiq

Fourny¹,

Dao²,

Cikis³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Lakehouse systems have reached in the past few years unprecedented size and heterogeneity and have been embraced by many industry players. However, they are often difficult to use as they lack the declarative language and optimization possibilities of relational engines. This paper introduces RumbleML, a high-level, declarative library integrated into the RumbleDB engine and with the JSONiq language. RumbleML allows using a single platform for data cleaning, data preparation, training, and inference, as well as management of models and results. It does it using a purely declarative language (JSONiq) for all these tasks and without any performance loss over existing platforms (e.g. Spark). The key insights of the design of RumbleML are that training sets, evaluation sets, and test sets can be represented as homogeneous sequences of flat objects; that models can be seamlessly embodied in function items mapping input test sets into prediction-augmented result sets; and that estimators can be seamlessly embodied in function items mapping input training sets to models. We argue that this makes JSONiq a viable and seamless programming language for data lakehouses across all their features, whether database-related or machine-learning-related. While lakehouses bring Machine Learning and Data Wrangling on the same platform, RumbleML also brings them to the same language, JSONiq. In the paper, we present the first prototype and compare its performance to Spark showing the benefit of a huge functionality and productivity gain for cleaning up, normalizing, validating data, feeding it into Machine Learning pipelines, and analyzing the output, all within the same system and language and at scale.

show abstract

Evaluating query languages and systems for high-energy physics data

Cited by 10 publications

References 47 publications

HL-LHC Analysis With ROOT: ROOT Project Input to the HL-LHC Computing Review (Stage 2)

HL-LHC Analysis With ROOT: ROOT Project Input to the HL-LHC Computing Review (Stage 2)

Leveraging state-of-the-art engines for large-scale data analysis in High Energy Physics

RumbleML: program the lakehouse with JSONiq

Contact Info

Product

Resources

About