Generating application-specific data layouts for in-memory databases

Abstract

Database applications are often developed with object-oriented languages while using relational databases as the backend. To accelerate these applications, developers would manually design customized data structures to store data in memory, and ways to utilize such data structures to answer queries. Doing so is brittle and requires a lot of effort. Alternatively, developers might automate the process by using relational physical design tools to create materialized views and indexes instead. However, the characteristics of object-oriented database applications are often distinct enough from traditional database applications such that classical relational query optimization techniques often cannot speed up queries that arise from such applications, as our experiments show. To address this, we build CHESTNUT, a data layout generator for in-memory object-oriented database applications. Given a memory budget, CHESTNUT generates customized in-memory data layouts and query plans to answer queries written using a subset of the Rails API, a common framework for building object-oriented database applications. CHESTNUT differs from traditional query optimizers and physical designers in two ways. First, CHESTNUT automatically generates data layouts that are customized for the application after analyzing their queries, hence CHESTNUT-generated data layouts are designed to be efficient to answer queries from such applications. Second, CHESTNUT uses a novel enumeration and verification-based algorithm to generate query plans that use such data layouts, rather than rule-based approaches as in traditional query optimizers. We evaluated CHESTNUT on four opensource Rails database applications. The result shows that it can reduce average query processing time by over 3.6× (and up to 42×), as compared to other in-memory relational database engines.

New Directions in Cloud Programming

“…We have designed a data structure synthesizer called Chestnut in our earlier work [76,88], focusing on database-backed web applications that are built using ORM libraries. Similar to HydroLogic, Chestnut takes in a user-provided data model and workload specification, and synthesizes data structures to store persistent data once it is loaded into memory.…”

Demonstration of Chestnut: An In-memory Data Layout Designer for Database Applications

“…It uses an external ILP solver to solve the ILP formulation and, once solved, generates C++ code for its chosen data layout and query plans. Our research paper [10] covers the technical details. The demo will be structured in the following way.…”

Deductive Optimization of Relational Data Storage

“…However, a compiler was never developed to create the layouts from this language; the paper demonstrated its point by implementing each layout by hand. Also related is Chestnut: a tool for generating specialized layouts for object queries [38]. Chestnut has separate layout and query languages and synthesizes a query after choosing the layout.…”