Qd-tree: Learning Data Layouts for Big Data Analytics

Abstract

Corporations today collect data at an unprecedented and accelerating scale, making the need to run queries on large datasets increasingly important. Technologies such as columnar block-based data organization and compression have become standard practice in most commercial database systems. However, the problem of best assigning records to data blocks on storage is still open. For example, today's systems usually partition data by arrival time into row groups, or range/hash partition the data based on selected fields. For a given workload, however, such techniques are unable to optimize for the important metric of the number of blocks accessed by a query. This metric directly relates to the I/O cost, and therefore performance, of most analytical queries. Further, they are unable to exploit additional available storage to drive this metric down further. In this paper, we propose a new framework called a querydata routing tree, or qd-tree, to address this problem, and propose two algorithms for their construction based on greedy and deep reinforcement learning techniques. Experiments over benchmark and real workloads show that a qd-tree can provide physical speedups of more than an order of magnitude compared to current blocking schemes, and can reach within 2× of the lower bound for data skipping based on selectivity, while providing complete semantic descriptions of created blocks.

Tsunami

“…We decided not to evaluate against these because Flood already showed consistent superiority over them [30]. We also do not evaluate against other learned multi-dimensional indexes because they are either optimized for disk [25,46] or optimize only based on the data distribution, not the query workload [9,44] (see §7).…”

“…Learning has also been applied to the challenge of reducing I/O cost for disk-based multi-dimensional indexes. Qd-tree [46] uses reinforcement learning to construct a partitioning strategy that minimizes the number of disk-based blocks accessed by a query. LISA [25] is a disk-based learned spatial index that achieves low storage consumption and I/O cost while supporting range queries, nearest neighbor queries, and insertions and deletions.…”

“…To support dynamic data, each leaf node in the Grid Tree could maintain a sibling node that acts as a delta index [39] in which inserts, updates, and deletes are buffered and periodically merged into the main node. Persistence Tsunami's techniques for reducing query skew and handling correlations are not restricted to in-memory scenarios and could be incorporated into an index for data resident on disk or SSD, perhaps by combining ideas from qd-tree [46] or LISA [25].…”

“…To address the shortcomings of traditional indexes, recent work has proposed the idea of learned multi-dimensional indexes [9,25,30,44,46]. In particular, Flood [30] is a in-memory multi-dimensional index that automatically optimizes its structure to achieve high performance on a particular dataset and workload.…”

A Survey on Deep Reinforcement Learning for Data Processing and Analytics

“…Notably, DRL incorporates deep learning (DL) techniques to handle complex unstructured data and has been designed to learn from historical data and self-exploration to solve notoriously hard and large-scale problems (e.g., AlphaGo [15]). In recent years, researchers from different communities have proposed DRL solutions to address issues in data processing and analytics [4], [16], [17]. We categorize existing works using DRL from two perspectives: system and application.…”