Real-Time LSM-Trees for HTAP Workloads

Abstract

Real-time data analytics systems such as SAP HANA, MemSQL, and IBM Wildfire employ hybrid data layouts, in which data are stored in different formats throughout their lifecycle. Recent data are stored in a row-oriented format to serve OLTP workloads and support high data rates, while older data are transformed to a column-oriented format for OLAP access patterns. We observe that a Log-Structured Merge (LSM) Tree is a natural fit for a lifecycleaware storage engine due to its high write throughput and leveloriented structure, in which records propagate from one level to the next over time. To build a lifecycle-aware storage engine using an LSM-Tree, we make a crucial modification to allow different data layouts in different levels, ranging from purely row-oriented to purely column-oriented, leading to a Real-Time LSM-Tree. We give a cost model and an algorithm to design a Real-Time LSM-Tree that is suitable for a given workload, followed by an experimental evaluation of LASER -a prototype implementation of our idea built on top of the RocksDB key-value store. In our evaluation, LASER is almost 5x faster than Postgres (a pure row-store) and two orders of magnitude faster than MonetDB (a pure column-store) for real-time data analytics workloads.

Hybrid Transactional/Analytical Processing Amplifies IO in LSM-Trees

“…ETL-based systems deliver decent performance on data analytics acting on reshaped data ingested from transaction engines but lose their real-time degree of data analytics mainly due to long delays in their ETL workflows. As more organiza-tions invest in real-time data analytics, recent years have seen commercial solutions [7]- [10] in industry and research outcomes [11]- [15] from academia; all of which evolve towards a single system capable of HTAP. The trend towards a single HTAP system requires the system to combine two types of data processing, despite their heterogeneous nature, into a single system.…”