Building Efficient Query Engines in a High-Level Language

Abstract

Abstraction without regret refers to the vision of using high-level programming languages for systems development without experiencing a negative impact on performance. A database system designed according to this vision offers both increased productivity and high performance, instead of sacrificing the former for the latter as is the case with existing, monolithic implementations that are hard to maintain and extend.In this article, we realize this vision in the domain of analytical query processing. We present LegoBase, a query engine written in the high-level programming language Scala. The key technique to regain efficiency is to apply generative programming: LegoBase performs source-to-source compilation and optimizes the entire query engine by converting the high-level Scala code to specialized, low-level C code. We show how generative programming allows to easily implement a wide spectrum of optimizations, such as introducing data partitioning or switching from a row to a column data layout, which are difficult to achieve with existing low-level query compilers that handle only queries. We demonstrate that sufficiently powerful abstractions are essential for dealing with the complexity of the optimization effort, shielding developers from compiler internals and decoupling individual optimizations from each other.We evaluate our approach with the TPC-H benchmark and show that: (a) With all optimizations enabled, our architecture significantly outperforms a commercial in-memory database as well as an existing query compiler. (b) Programmers need to provide just a few hundred lines of high-level code for implementing the optimizations, instead of complicated low-level code that is required by existing query compilation approaches. (c) These optimizations may potentially come at the cost of using more system memory for improved performance. (d) The compilation overhead is low compared to the overall execution time, thus making our approach usable in practice for compiling query engines.

Multi-layer optimizations for end-to-end data analytics

“…Similar to IFAQ, DBToaster [31] uses a bag-based collection, yet for incremental view maintenance. A key challenge for such collection-based languages is efficient code generation, which has been investigated both by the DB [39,52,53] and PL [28,33,49] communities. The IFAQ languages are also inspired by the same model, with more similarities to monoid comprehension; the summation operator can be considered as a monoid comprehension.…”

Learning Models over Relational Data: A Brief Tutorial

“…Code Specialization and Query Compilation It involves generating code specific to the query and the schema of its input data, following prior work [36,52,53], and also specific to the model to be learned. This technique improves the runtime performance by inlining code and improving cache locality for the hot data path.…”

Push versus pull-based loop fusion in query engines

“…implementing inter-operator parallelism in query engines. In the context of query compilation, stream processing engines such as StreamBase (Tibbetts et al, 2011) and Spade (Gedik et al, 2008), as well as HyPer (Neumann, 2011) and LegoBase (Klonatos et al, 2014a;Shaikhha et al, 2018) use a push-based query engine approach.…”

“…The absolute execution times for all these queries can be found in Table 5. Note that, the performance difference with LegoBase (Klonatos et al, 2014a;Shaikhha et al, 2018) and DBLAB/LB (Shaikhha et al, 2016) is due to the lack of additional optimizations provided by these systems such as data-structure specialization.…”