Building efficient query engines in a high-level language

Abstract

In this paper we advocate that it is time for a radical rethinking of database systems design. Developers should be able to leverage high-level programming languages without having to pay a price in efficiency. To realize our vision of abstraction without regret, we present LegoBase, a query engine written in the high-level programming language Scala. The key technique to regain efficiency is to apply generative programming: the Scala code that constitutes the query engine, despite its high-level appearance, is actually a program generator that emits specialized, low-level C code. We show how the combination of high-level and generative programming allows to easily implement a wide spectrum of optimizations that are difficult to achieve with existing low-level query compilers, and how it can continuously optimize the query engine.We evaluate our approach with the TPC-H benchmark and show that: (a) with all optimizations enabled, our architecture significantly outperforms a commercial in-memory database system as well as an existing query compiler, (b) these performance improvements require programming just a few hundred lines of high-level code instead of complicated low-level code that is required by existing query compilers and, finally, that (c) the compilation overhead is low compared to the overall execution time, thus making our approach usable in practice for efficiently compiling query engines.

Fast scans on key-value stores

“…When transforming a batch of scan requests into a single shared scan, TellStore uses just-in-time compilation with LLVM to get highly-tuned machine code. In essence, TellStore combines the batch-oriented shared scan technique of [22] with LLVM code generation described in [33,27].…”

Extensible, Fast And Secure Scala Expression Evaluation Engine

“…Another example of research done on building eficient query engines is presented in [28]. What constitutes the research particularly interesting for our work is Scala language that has been selected to implement LegoBase -a query engine being analysed by authors.…”

Fast queries over heterogeneous data through engine customization

“…Runtime code generation has become an established mechanism, used by several relational engines [6,34,36,43,45,49]. HIQUE [36] generates cache-conscious code via code templates.…”

“…HyPer [43] uses the LLVM compiler [37] to generate low-level machine code. LegoBase [34] goes through numerous rewriting ("staging") steps to generate C code. Proteus follows the HyPer paradigm and relies on LLVM too.…”