Design and evaluation of storage organizations for read-optimized main memory databases

Chasseur, Craig; Patel, Jignesh M.

doi:10.14778/2536258.2536260

Cited by 21 publications

(16 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unlike AMIE, it works on facts extracted from free text that are not mapped to crisp relations. QuickFOIL [44] is a standard ILP system based on a generic top-down greedy algorithm and implemented on top of the QuickStep in-memory storage engine [7]. It learns a set of hypotheses (Horn rules) from positive and negative examples of a target relation and a collection of background facts.…”

Section: Logical Rule Miningmentioning

confidence: 99%

Fast rule mining in ontological knowledge bases with AMIE $$+$$ +

et al. 2015

View full text Add to dashboard Cite

Recent advances in information extraction have led to huge knowledge bases (KBs), which capture knowledge in a machine-readable format. Inductive Logic Programming (ILP) can be used to mine logical rules from these KBs, such as "If two persons are married, then they (usually) live in the same city". While ILP is a mature field, mining logical rules from KBs is difficult, because KBs make an open world assumption. This means that absent information cannot be taken as counterexamples. Our approach AMIE [16] has shown how rules can be mined effectively from KBs even in the absence of counterexamples. In this paper, we show how this approach can be optimized to mine even larger KBs with more than 12M statements. Extensive experiments show how our new approach, AMIE+, extends to areas of mining that were previously beyond reach.

show abstract

Section: Logical Rule Miningmentioning

confidence: 99%

Fast rule mining in ontological knowledge bases with AMIE $$+$$ +

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Au-toAdmin [1,2,33,36,48,52] optimizes a database and its physical design using machine learning and data mining. Some systems perform partitioning based on workload access patterns [2,3,9], while other systems are based on graph-based workload modeling techniques [10,40,44]. Sun et al [45,46] (discussed in this paper) extract features from each workload operation based on its predicates.…”

Section: Related Workmentioning

confidence: 99%

Qd-tree: Learning Data Layouts for Big Data Analytics

Yang

Chandramouli

Wang

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Corporations today collect data at an unprecedented and accelerating scale, making the need to run queries on large datasets increasingly important. Technologies such as columnar block-based data organization and compression have become standard practice in most commercial database systems. However, the problem of best assigning records to data blocks on storage is still open. For example, today's systems usually partition data by arrival time into row groups, or range/hash partition the data based on selected fields. For a given workload, however, such techniques are unable to optimize for the important metric of the number of blocks accessed by a query. This metric directly relates to the I/O cost, and therefore performance, of most analytical queries. Further, they are unable to exploit additional available storage to drive this metric down further. In this paper, we propose a new framework called a querydata routing tree, or qd-tree, to address this problem, and propose two algorithms for their construction based on greedy and deep reinforcement learning techniques. Experiments over benchmark and real workloads show that a qd-tree can provide physical speedups of more than an order of magnitude compared to current blocking schemes, and can reach within 2× of the lower bound for data skipping based on selectivity, while providing complete semantic descriptions of created blocks.

show abstract

“…Data partitioning covers both the problem of partitioning a relation across multiple servers and within a single server [63,79,80]. Partitioning across both rows and columns is introduced by several systems to account for different read access patterns (e.g., on fact tables and dimension tables) [4,11,26]. The workload is frequently modeled as a graph [29,69,75].…”

Section: Related Workmentioning

confidence: 99%

Optimal column layout for hybrid workloads

2019

View full text Add to dashboard Cite

Data-intensive analytical applications need to support both efficient reads and writes. However, what is usually a good data layout for an update-heavy workload, is not well-suited for a read-mostly one and vice versa. Modern analytical data systems rely on columnar layouts and employ delta stores to inject new data and updates. We show that for hybrid workloads we can achieve close to one order of magnitude better performance by tailoring the column layout design to the data and query workload. Our approach navigates the possible design space of the physical layout: it organizes each column's data by determining the number of partitions, their corresponding sizes and ranges, and the amount of buffer space and how it is allocated. We frame these design decisions as an optimization problem that, given workload knowledge and performance requirements, provides an optimal physical layout for the workload at hand. To evaluate this work, we build an in-memory storage engine, Casper, and we show that it outperforms state-of-the-art data layouts of analytical systems for hybrid workloads. Casper delivers up to 2.32× higher throughput for update-intensive workloads and up to 2.14× higher throughput for hybrid workloads. We further show how to make data layout decisions robust to workload variation by carefully selecting the input of the optimization.

show abstract

Design and evaluation of storage organizations for read-optimized main memory databases

Cited by 21 publications

References 23 publications

Fast rule mining in ontological knowledge bases with AMIE $$+$$ +

Fast rule mining in ontological knowledge bases with AMIE $$+$$ +

Qd-tree: Learning Data Layouts for Big Data Analytics

Optimal column layout for hybrid workloads

Contact Info

Product

Resources

About