Storing and processing temporal data in a main memory column store

Abstract

Managing and accessing temporal data is of increasing importance in industry. So far, most companies model the time dimension on the application layer rather than pushing down the operators to the database, which leads to a significant performance overhead. The goal of this PhD thesis is to develop a native support of temporal features for SAP HANA, which is a commercial inmemory column store database system. We investigate different alternatives to store temporal data physically and analyze the tradeoffs arising from different memory layouts which cluster the data either by time or by space dimension. Taking into account the underlying physical representation, different temporal operators such as temporal aggregation, time travel and temporal join have to be executed efficiently. We present a novel data structure called Timeline Index and algorithms based on this index, which have a very competitive performance for all temporal operators beating existing best-of-breed approaches by factors, sometimes even by orders of magnitude. The results of this thesis are currently being integrated into HANA, with the goal of being shipped to the customers as a productive release within the next few months.

In-Memory Big Data Management and Processing: A Survey

“…HANA supports temporal queries, such as temporal aggregation, time travel and temporal join, based on a unified index structure called the Timeline Index [88], [242], [247]. For every logical table, HANA keeps the current version of the table in a Current Table and the whole history of previous versions in a Temporal Table, accompanied with a Timeline Index to facilitate temporal queries.…”

“…In-memory data layouts have a significant impact on the memory usage and cache utilization. Columnar layout of relational table facilitates scan-like queries/analytics as it can achieve good cache locality [41], [88], and can achieve better data compression [89], but is not optimal for OLTP queries that need to operate on the row level [74], [90]. It is also possible to have a hybrid of row and column layouts, such as PAX which organizes data by columns only within a page [91], and SAP HANA with multilayer stores consisting of several delta row/column stores and a main column store, which are merged periodically [74].…”

Cache-efficient sweeping-based interval joins for extended Allen relation predicates

“…Temporal data are found in many financial, business, and scientific applications running on top of database management systems (DBMSs), i.e., supporting these applications through efficient temporal operator implementations is crucial. For example, Kaufmann states that there are several temporal queries in the hundred most expensive queries executed on SAP ERP [23], many of which have to be implemented in the application layer, as the underlying infrastructure does not directly support the processing of temporal data. According to [23], customers of SAP desperately need (advanced) temporal operators for efficiently running queries pertaining to legal, compliance, and auditing processes.…”

Cache-Efficient Sweeping-Based Interval Joins for Extended Allen Relation Predicates (Extended Version)

“…Temporal data is found in many financial, business, and scientific applications running on top of database management systems (DBMSs), i.e., supporting these applications through efficient temporal operator implementations is crucial. For example, Kaufmann states that there are several temporal queries in the hundred most expensive queries executed on SAP ERP [23], many of which have to be implemented in the application layer, as the underlying infrastructure does not directly support the processing of temporal data. According to [23], customers of SAP desperately need (advanced) temporal operators for efficiently running queries pertaining to legal, compliance, and auditing processes.…”