Optimal column layout for hybrid workloads

Abstract

Data-intensive analytical applications need to support both efficient reads and writes. However, what is usually a good data layout for an update-heavy workload, is not well-suited for a read-mostly one and vice versa. Modern analytical data systems rely on columnar layouts and employ delta stores to inject new data and updates. We show that for hybrid workloads we can achieve close to one order of magnitude better performance by tailoring the column layout design to the data and query workload. Our approach navigates the possible design space of the physical layout: it organizes each column's data by determining the number of partitions, their corresponding sizes and ranges, and the amount of buffer space and how it is allocated. We frame these design decisions as an optimization problem that, given workload knowledge and performance requirements, provides an optimal physical layout for the workload at hand. To evaluate this work, we build an in-memory storage engine, Casper, and we show that it outperforms state-of-the-art data layouts of analytical systems for hybrid workloads. Casper delivers up to 2.32× higher throughput for update-intensive workloads and up to 2.14× higher throughput for hybrid workloads. We further show how to make data layout decisions robust to workload variation by carefully selecting the input of the optimization.

OLxPBench: Real-time, Semantically Consistent, and Domain-specific are Essential in Benchmarking, Designing, and Implementing HTAP Systems

“…The one contains intricate transactions and queries, such as CH-benCHmark [41], CBTR [26], and HTAPBench [42]. The other includes a mix of simple insert/select operations, i.e., ADAPT [43] and HAP [44]. The real-time queries generally involve simple aggregate operations and the analytical queries include more complex operations.…”

Real-Time LSM-Trees for HTAP Workloads

“…Workload: We generate workloads using the benchmark proposed by previous work [11,12]. The benchmark consists of the following queries that are common in HTAP workloads: (𝑄 1 ) inserts new tuples, (𝑄 2 ) is a point query that selects a specific row, (𝑄 3 ) is an aggregate query that computes the maximum values of selected attributes over selected tuples, (𝑄 4 ) is an arithmetic query that sums a subset of attributes over the selected tuples, and (𝑄 5 ) is an update query that updates a subset of attributes of a specific row.…”

“…Along with the inserts, we issue 100 updates per second, i.e., one percent of the insert rate, via 𝑄 5 , where a randomly chosen column value is updated for a recently inserted key. This update pattern mimics updates and corrections frequently taking place in mixed analytical and transactional processing [12]. Furthermore, we control the access patterns throughout the data lifecycle by selecting 𝑘, 𝑣, 𝑣 𝑠 , and 𝑣 𝑒 for queries 𝑄 2 − 𝑄 4 such that the upper levels of the LSM-Tree are mostly accessed by point read operations and wider projections, whereas lower levels are accessed by scan operations and narrower projections.…”

Endure: A Robust Tuning Paradigm for LSM Trees Under Workload Uncertainty

“…These tools can be broadly classified as offline workload analysis for index and views design [2,3,22,26,84,93], and periodic online workload analysis [18,[75][76][77] to capture workload drift [43]. In addition, there has been research on reducing the magnitude of the search space of tuning [17,27] and on deciding the optional data partitioning [9,65,79,81,82]. These approaches assume that the input information about resources and workload is accurate.…”