Emerging Database Technologies and Their Applicability to High Energy Physics: A First Look at SciDB

Abstract

Abstract. Traditional relational databases have not always been well matched to the needs of data-intensive sciences, and to the needs of high energy physics data stores in particular. To address this mismatch, members of the database community and people involved with large scientific data stores in a variety of disciplines have inaugurated an open-source project, SciDB, that aims to develop and deliver database technologies suited to the needs of dataintensive sciences. This paper describes early experience using the first release of SciDB with an initial subset of high energy physics data structures and query patterns. It examines the early capabilities of SciDB, and describes requirements that further development must address if emerging database technologies such as SciDB are to accommodate the data structures, query patterns, computations, and use cases of high energy physics.

From Static to Agile - Interactive Particle Physics Analysis in the SAP HANA DB

“…However, particle physics data does not have an array-like structure that requires element locality, but are rather a huge set of many independent measurements, hence, our analysis would not benefit from using an array-based DBMS. Although the authors of a related project come to a similar conclusion (Malon et al, 2011), there have been efforts to utilize SciDB for the ATLAS Tag Database (Malon et al, 2012).…”

“…Particle Physics and Database Systems. The AT-LAS Tag Database (Cranshaw et al, 2008;Malon et al, 2011;Malon et al, 2012) stores event-level metadata in a relational database system, so that scientists can preselect events according to their analysis criteria by relational means. Based on the preselection, the analysis is then run conventionally using C++ programs on ROOT files, but only using the selected raw data files rather than on the whole set.…”

“…The idea of combining customized in-situ processing on raw data files with the advantages of the columnar data processing capabilities of a modern DBMS has already been mentioned in (Malon et al, 2011). A similar -"NoDB" -approach was implemented in the work of Karpathiotakis et.…”

Evaluating query languages and systems for high-energy physics data

“…The latter attribute type denormalizes the data into non-first normal form (NF 2 ). Such a data representation was one of the reasons why SQL-based query engines initially found little adoption in the HEP community for analysis use-cases -in addition to the lack of support for User-Defined Functions (UDF) [4]-, as early engines did not provide support for nested data types. The database community has since made major strides, and many relational database systems provide mature support for processing nested attributes and defining custom user logic.…”