Column imprints

Abstract

Large scale data warehouses rely heavily on secondary indexes, such as bitmaps and b-trees, to limit access to slow IO devices. However, with the advent of large main memory systems, cache conscious secondary indexes are needed to improve also the transfer bandwidth between memory and cpu. In this paper, we introduce column imprint, a simple but efficient cache conscious secondary index. A column imprint is a collection of many small bit vectors, each indexing the data points of a single cacheline. An imprint is used during query evaluation to limit data access and thus minimize memory traffic. The compression for imprints is cpu friendly and exploits the empirical observation that data often exhibits local clustering or partial ordering as a side-effect of the construction process. Most importantly, column imprint compression remains effective and robust even in the case of unclustered data, while other state-of-the-art solutions fail. We conducted an extensive experimental evaluation to assess the applicability and the performance impact of the column imprints. The storage overhead, when experimenting with real world datasets, is just a few percent over the size of the columns being indexed. The evaluation time for over 40000 range queries of varying selectivity revealed the efficiency of the proposed index compared to zonemaps and bitmaps with WAH compression.

Scaling column imprints using advanced vectorization

“…To evaluate the performance of the vectorized version of imprints, we used a subset of the collection of datasets as in the original paper of Column Imprints [7]. e data sets consist of 6,476 di erent columns, with a maximum number of records of 600 millions, and contain integer and decimal types of various length.…”

“…For a complete presentation of Column Imprints and a detailed explanation of each algorithm we refer the reader to [7].…”

“…For completeness of the presentation in this paper, we include the time dominant code snippets of the scalar implementation of the imprints algorithms, as identi ed in the previous section and described in details in [7]. is work is about substituting these critical snippets of code in the imprints creation and querying algorithms in order to enable advanced vectorization optimizations.…”

“…Column Imprints [7] is a secondary index for answering range queries in a read optimized columnar database. Imprints are preltering bit-vectors that quickly determine if a cache line or block of data contains values that satisfy the range predicates of a query.…”

Benchmarking and improving point cloud data management in MonetDB

“…a column-store database kernel, (b.) CPU-tuned and automatically parallelized query execution architecture, (c.) a novel indexing strategy called Imprints [14], that creates index structures on-the-fly when the columns are queried, (d.) completely modifiable and extendable system thanks to the liberal open-source license, (e.) run time query optimization, (f.) a modular software architecture, (g.) a basis on the SQL 2003 standard with full support for foreign keys, joins, views, triggers and stored procedures, and (h.) being a fully ACID compliant system that supports a rich spectrum of programming interfaces (JDBC, ODBC, PHP, Python, RoR, C/C++, Perl).…”