Code generation for efficient query processing in managed runtimes

Nagel, Fabian; Bierman, Gavin; Viglas, Stratis D.

doi:10.14778/2732977.2732984

Cited by 22 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In case of a local aggregation, we optimize the number of hash tables (1,8,64,256,1024,16384,65536) as a multiplier of the number of maximal compute units of the OpenCL device to test different levels of thread over-subscription. We also optimize the number of threads per hash table (16,32,64,128,256,512,1024) to find the best configuration between high parallelism and synchronization overhead [7x7 variants]. In case of global aggregation, we optimize the number of threads per hash table, which configurations are identical to local aggregation [7 variants].…”

Section: Variant Space Of Generated Variantsmentioning

confidence: 99%

Generating custom code for efficient query execution on heterogeneous processors

et al. 2018

View full text Add to dashboard Cite

Processor manufacturers build increasingly specialized processors to mitigate the effects of the power wall to deliver improved performance. Currently, database engines are manually optimized for each processor: A costly and error prone process.In this paper, we propose concepts to enable the database engine to perform per-processor optimization automatically. Our core idea is to create variants of generated code and to learn a fast variant for each processor. We create variants by modifying parallelization strategies, specializing data structures, and applying different code transformations.Our experimental results show that the performance of variants may diverge up to two orders of magnitude. Therefore, we need to generate custom code for each processor to achieve peak performance. We show that our approach finds a fast custom variant for multi-core CPUs, GPUs, and MICs.

show abstract

Section: Variant Space Of Generated Variantsmentioning

confidence: 99%

Generating custom code for efficient query execution on heterogeneous processors

et al. 2018

View full text Add to dashboard Cite

show abstract

“…With the advent of in-memory DBMSes, the code layout of the instructions becomes a very important factor. As a result, query compilation uses code generation and compilation techniques in order to inline virtual functions and further specialize the code to improve cache locality [20,2,32,35,42,33,34,30,53,12,41,29,3,48,28]. As a result of that, the code pattern used in each pipelining algorithm really matters.…”

Section: Compiled Enginesmentioning

confidence: 99%

Push versus pull-based loop fusion in query engines

Shaikhha¹,

Dashti²,

Koch³

2018

J. Funct. Prog.

View full text Add to dashboard Cite

Database query engines use pull-based or push-based approaches to avoid the materialization of data across query operators. In this paper, we study these two types of query engines in depth and present the limitations and advantages of each engine. Similarly, the programming languages community has developed loop fusion techniques to remove intermediate collections in the context of collection programming. We draw parallels between the DB and PL communities by demonstrating the connection between pipelined query engines and loop fusion techniques. Based on this connection, we propose a new type of pull-based engine, inspired by a loop fusion technique, which combines the benefits of both approaches. Then we experimentally evaluate the various engines, in the context of query compilation, for the first time in a fair environment, eliminating the biasing impact of ancillary optimizations that have traditionally only been used with one of the approaches. We show that for realistic analytical workloads, there is no considerable advantage for either form of pipelined query engine, as opposed to what recent research suggests. Also, by using microbenchmarks we show that our proposed engine dominates the existing engines by combining the benefits of both.

show abstract

“…Automatic just-in-time compilation has now become a standard technique [14,25,30,31] also used by other database systems (e.g., HyPer [24,35]). Indeed, our group has also implemented a query-compiler on its own which is part of a commercial product however without a published description of the details.…”

Section: Related Workmentioning

confidence: 99%

Fast OLAP query execution in main memory on large data in a cluster

Weidner

Dees

Sanders

2013

2013 IEEE International Conference on Big Data

View full text Add to dashboard Cite

Main memory column-stores have proven to be efficient for processing analytical queries. Still, there has been much less work in the context of clusters. Using only a single machine poses several restrictions: Processing power and data volume are bounded to the number of cores and main memory fitting on one tightly coupled system. To enable the processing of larger data sets, switching to a cluster becomes necessary. In this work, we explore techniques for efficient execution of analytical SQL queries on large amounts of data in a parallel database cluster while making maximal use of the available hardware. This includes precompiled query plans for efficient CPU utilization, full parallelization on single nodes and across the cluster, and efficient inter-node communication. We implement all features in a prototype for running a subset of TPC-H benchmark queries. We evaluate our implementation using a 128 node cluster running TPC-H queries with 30 000 gigabyte of uncompressed data.

show abstract

Code generation for efficient query processing in managed runtimes

Cited by 22 publications

References 22 publications

Generating custom code for efficient query execution on heterogeneous processors

Generating custom code for efficient query execution on heterogeneous processors

Push versus pull-based loop fusion in query engines

Fast OLAP query execution in main memory on large data in a cluster

Contact Info

Product

Resources

About