LeanStore: In-Memory Data Management beyond Main Memory

Leis, Viktor; Haubenschild, Michael; Kemper, Alfons; Neumann, Thomas

doi:10.1109/icde.2018.00026

Cited by 67 publications

(14 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[12] propose a design that uses pointer swizzling to eliminate buffer pool overheads for memory resident data, in which raw pointers to loaded pieces are kept as reference to avoid lookup in separate queues managed by the buffer pool. LeanStore [17] uses pointer swizzling and replacement strategy by identifying infrequently accessed pages and an epoch-based technique to pin pages and achieve performance comparable to main memory systems. In HANA's NSE, the read optimized section of a hybrid column benefits from the NSE architecture and is managed by NSE's buffer cache for better tuning and optimizations that are only applicable to paged resources.…”

Section: Related Workmentioning

confidence: 99%

Native store extension for SAP HANA

et al. 2019

View full text Add to dashboard Cite

We present an overview of SAP HANA's Native Store Extension (NSE). This extension substantially increases database capacity, allowing to scale far beyond available system memory. NSE is based on a hybrid in-memory and paged column store architecture composed from data access primitives. These primitives enable the processing of hybrid columns using the same algorithms optimized for traditional HANA's in-memory columns. Using only three key primitives, we fabricated byte-compatible counterparts for complex memory resident data structures (e.g. dictionary and hash-index), compressed schemes (e.g. sparse and run-length encoding), and exotic data types (e.g. geo-spatial). We developed a new buffer cache which optimizes the management of paged resources by smart strategies sensitive to page type and access patterns. The buffer cache integrates with HANA's new execution engine that issues pipelined prefetch requests to improve disk access patterns. A novel load unit configuration, along with a unified persistence format, allows the hybrid column store to dynamically switch between inmemory and paged data access to balance performance and storage economy according to application demands while reducing Total Cost of Ownership (TCO). A new partitioning scheme supports load unit specification at table, partition, and column level. Finally, a new advisor recommends optimal load unit configurations. Our experiments illustrate the performance and memory footprint improvements on typical customer scenarios.

show abstract

Section: Related Workmentioning

confidence: 99%

Native store extension for SAP HANA

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Because SSDs provide block-wise access, and data needs to be loaded into DRAM before it can be processed, a number of high-performance storage engines have recently been proposed to exploit fast SSDs [7,31,39,50]. Using techniques like pointer swizzling [31] and scalable optimistic synchronization [32,33], systems like LeanStore [31] offer transparent buffer management with very little overhead. This SSD-optimized approach is a new database architecture that significantly differs from both the disk-based and the in-memory designs.…”

Section: Introductionmentioning

confidence: 99%

“…It extends the scalable logging scheme proposed by Wang and Johnson [52] with a novel continuous checkpointing algorithm, an efficient page provisioning strategy, and an optimized cross-log commit protocol. We integrate all these components into LeanStore [31], a lightweight buffer manager that uses pointer swizzling and optimistic synchronization. The resulting system can sustain transaction rates close to that of a pure in-memory system when all data fits in DRAM, as well as handle out-of-memory workloads transparently.…”

Section: Introductionmentioning

confidence: 99%

Rethinking Logging, Checkpoints, and Recovery for High-Performance Storage Engines

Haubenschild

Sauer

Neumann

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

For decades, ARIES has been the standard for logging and recovery in database systems. ARIES offers important features like support for arbitrary workloads, fuzzy checkpoints, and transparent index recovery. Nevertheless, many modern inmemory database systems use more lightweight approaches that have less overhead and better multi-core scalability but only work well for the in-memory setting. Recently, a new class of high-performance storage engines has emerged, which exploit fast SSDs to achieve performance close to pure in-memory systems but also allow out-of-memory workloads. For these systems, ARIES is too slow whereas inmemory logging proposals are not applicable. In this work, we propose a new logging and recovery design that supports incremental and fuzzy checkpointing, index recovery, out-of-memory workloads, and low-latency transaction commits. Our continuous checkpointing algorithm guarantees bounded recovery time. Using per-thread logging with minimal synchronization, our implementation achieves near-linear scalability on multi-core CPUs. We implemented and evaluated these techniques in our LeanStore storage engine. For working sets that fit in main memory, we achieve performance close to that of an in-memory approach, even with logging, checkpointing, and dirty page writing enabled. For the out-of-memory scenario, we outperform a state-of-the-art ARIES implementation by a factor of two.

show abstract

“…The B+tree is organized such that nodes are represented by pages, which are the unit of data movement and buffering. Optimizations like pointer swizzling [8] and low-overhead replacement policies [10] may apply. The atomicity and durability of writes to pages buffered in DRAM is guaranteed by write-ahead logging (WAL).…”

Section: Introductionmentioning

confidence: 99%

Persistent Buffer Management with Optimistic Consistency

Lersch

Lehner

Oukid

2019

Proceedings of the 15th International Workshop on Data Management on New Hardware

View full text Add to dashboard Cite

Finding the best way to leverage non-volatile memory (NVM) on modern database systems is still an open problem. The answer is far from trivial since the clear boundary between memory and storage present in most systems seems to be incompatible with the intrinsic memory-storage duality of NVM. Rather than treating NVM either solely as memory or solely as storage, in this work we propose how NVM can be simultaneously used as both in the context of modern database systems. We design a persistent buffer pool on NVM, enabling pages to be directly read/written by the CPU (like memory) while recovering corrupted pages after a failure (like storage). The main benefits of our approach are an easy integration in the existing database architectures, reduced costs (by replacing DRAM with NVM), and faster peak-performance recovery.

show abstract

LeanStore: In-Memory Data Management beyond Main Memory

Cited by 67 publications

References 28 publications

Native store extension for SAP HANA

Native store extension for SAP HANA

Rethinking Logging, Checkpoints, and Recovery for High-Performance Storage Engines

Persistent Buffer Management with Optimistic Consistency

Contact Info

Product

Resources

About