On the Impact of Memory Allocation on High-Performance Query Processing

Durner, Dominik; Leis, Viktor; Neumann, Thomas

doi:10.1145/3329785.3329918

Cited by 23 publications

(15 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This way, we can control the expected arrival rate of 𝜆 queries per second. This is similar to [2] -by not spacing the queries uniformly, we obtain dynamic workload fluctuations with bursts of queries arriving in short succession. To obtain a mixture of long and short running analytical requests, we sample from TPC-H queries at SF3 and SF30.…”

Section: Methodssupporting

confidence: 54%

See 1 more Smart Citation

Self-Tuning Query Scheduling for Analytical Workloads

Wagner

Kohn

Neumann

2021

Proceedings of the 2021 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

Most database systems delegate scheduling decisions to the operating system. While such an approach simplifies the overall database design, it also entails problems. Adaptive resource allocation becomes hard in the face of concurrent queries. Furthermore, incorporating domain knowledge to improve query scheduling is difficult.To mitigate these problems, many modern systems employ forms of task-based parallelism. The execution of a single query is broken up into small, independent chunks of work (tasks). Now, fine-grained scheduling decisions based on these tasks are the responsibility of the database system. Despite being commonplace, little work has focused on the opportunities arising from this execution model.In this paper, we show how task-based scheduling in database systems opens up new areas for optimization. We present a novel lock-free, self-tuning stride scheduler that optimizes query latencies for analytical workloads. By adaptively managing query priorities and task granularity, we provide high scheduling elasticity. By incorporating domain knowledge into the scheduling decisions, our system is able to cope with workloads that other systems struggle with. Even at high load, we retain near optimal latencies for short running queries. Compared to traditional database systems, our design often improves tail latencies by more than 10x. CCS CONCEPTS• Information systems → Online analytical processing engines; Autonomous database administration.

show abstract

Section: Methodssupporting

confidence: 54%

“…In a more broad context, our work on scheduling falls into the important area of workload management within database systems. This is a wide ranging field incorporating topics like workload classification, admission control and resource management for memory and I/O [2,3,13,18,19].…”

Section: Related Workmentioning

confidence: 99%

Self-Tuning Query Scheduling for Analytical Workloads

Wagner

Kohn

Neumann

2021

Proceedings of the 2021 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several prior studies have measured the effects of various basis factors on database performance. A recent study found that a good memory allocator alone can improve analytical query processing performance by 2.7× [14]. A separate study presented a detailed evaluation of implementation and design choices in main-memory database systems, with a heavy focus on MVCC [55].…”

Section: Basis Factorsmentioning

confidence: 99%

Opportunities for optimism in contended main-memory multicore transactions

et al. 2020

View full text Add to dashboard Cite

Optimistic concurrency control, or OCC, can achieve excellent performance on uncontended workloads for main-memory transactional databases. Contention causes OCC's performance to degrade, however, and recent concurrency control designs, such as hybrid OCC/locking systems and variations on multiversion concurrency control (MVCC), have claimed to outperform the best OCC systems. We evaluate several concurrency control designs under varying contention and varying workloads, including TPCC, and find that implementation choices unrelated to concurrency control may explain much of OCC's previously-reported degradation. When these implementation choices are made sensibly, OCC performance does not collapse on high-contention TPC-C. We also present two optimization techniques, commit-time updates and timestamp splitting , that can dramatically improve the high-contention performance of both OCC and MVCC. Though these techniques are known, we apply them in a new context and highlight their potency: when combined, they lead to performance gains of 3.4X for MVCC and 3.6X for OCC in a TPC-C workload.

show abstract

“…Updating the metadata (e.g., bitmaps and logs) in the header space requires small writes to persistent memory. To study its access pattern, we profile the memory addresses of the first 1000 flush operations of metadata when running the DBMStest benchmark [16] using 4 allocators including nvm_malloc, PAllocator, PMDK, and Makalu. We show the results in Figure 2.…”

Section: Allocator-induced Small Random Accessmentioning

confidence: 99%

NVAlloc: rethinking heap metadata management in persistent memory allocators

Dong

Hong

et al. 2022

Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

View full text Add to dashboard Cite

Persistent memory allocation is a fundamental building block for developing high-performance and in-memory applications. Existing persistent memory allocators suffer from suboptimal heap organizations that introduce repeated cache line flushes and small random accesses in persistent memory. Worse, many allocators use static slab segregation resulting in a dramatic increase in memory consumption when allocation request size is changed. In this paper, we design a novel allocator, named NVAlloc, to solve the above issues simultaneously. First, NVAlloc eliminates cache line reflushes by mapping contiguous data blocks in slabs to interleaved metadata entries stored in different cache lines. Second, it writes small metadata units to a persistent bookkeeping log in a sequential pattern to remove random heap metadata accesses in persistent memory. Third, instead of using static slab segregation, it supports slab morphing, which allows slabs to be transformed between size classes to significantly improve slab usage. NVAlloc is complementary to the existing consistency models. Results on 6 benchmarks demonstrate that NVAlloc improves the performance of state-of-the-art persistent memory allocators by up to 6.4x and 57x for small and large allocations, respectively. Using NVAlloc reduces memory usage by up to 57.8%. Besides, we integrate NVAlloc in a persistent FPTree. Compared to the state-of-the-art allocators, NVAlloc improves the performance of this application by up to 3.1x. CCS CONCEPTS• Software and its engineering → Allocation / deallocation strategies; • Hardware → Non-volatile memory.

show abstract

On the Impact of Memory Allocation on High-Performance Query Processing

Cited by 23 publications

References 12 publications

Self-Tuning Query Scheduling for Analytical Workloads

Self-Tuning Query Scheduling for Analytical Workloads

Opportunities for optimism in contended main-memory multicore transactions

NVAlloc: rethinking heap metadata management in persistent memory allocators

Contact Info

Product

Resources

About