Leveraging compression in the tableau data engine

Abstract

Data sets are growing rapidly and there is an attendant need for tools that facilitate human analysis of them in a timely manner. To help meet this need, column-oriented databases (or "column stores") have come into wide use because of their low latency on analytic workloads. Column stores use a number of techniques to produce these dramatic performance techniques, including the ability to perform operations directly on compressed data.In this paper, we describe how the Tableau Data Engine (an internally developed column store) leverages a number of compression techniques to improve query performance. The approach is simpler than existing systems for operating on compressed data and more unified, removing the necessity for custom data access mechanisms. The approach also uses some novel metadata extraction techniques to improve the choices made by the system's run-time optimizer.

On Improving User Response Times in Tableau

“…This section gives a quick overview of one specific technique and is largely a summary of Sect. 5.2 from [2].…”

“…It has been described in [1] and [2]. Most features described in the above papers have been shipped before Tableau 9.0, except for the new performance improvements covered in Sect.…”

“…Such techniques have been discussed in [2]. The implementation has now become part of the Tableau 9.0 release.…”

Hillview

“…Several systems follow this approach. Microsoft PowerBI [67] using Di-rectQuery [26] and Polaris/Tableau [90,98,99] provide plug-ins to many analytics engines; as discussed in [17], the users of such systems have to carefully avoid many queries that cannot be answered efficiently. IBM BigSheets [12] computes interactively only over a subset of the data; once the user settles on a query, it is actually run in batch mode using Spark.…”

Multi-core column-store parallelization under concurrent workload

“…During inter-query parallelization, to maximize multi-core utilization multiple such queries are executed concurrently. On the other hand, most systems such as MonetDB, Vectorwise, Tableau, and SQL Server [7,30] use intra-query parallelization, using the exchange operator [16], where a single query executes on multiple cores. We use the following setup to understand which technique performs better.…”