Energy Analysis of Hardware and Software Range Partitioning

Abstract

Data partitioning is a critical operation for manipulating large datasets because it subdivides tasks into pieces that are more amenable to efficient processing. It is often the limiting factor in database performance and represents a significant fraction of the overall runtime of large data queries. This article measures the performance and energy of state-of-the-art software partitioners, and describes and evaluates a hardware range partitioner that further improves efficiency.The software implementation is broken into two phases, allowing separate analysis of the partition function computation and data shuffling costs. Although range partitioning is commonly thought to be more expensive than simpler strategies such as hash partitioning, our measurements indicate that careful data movement and optimization of the partition function can allow it to approach the throughput and energy consumption of hash or radix partitioning.For further acceleration, we describe a hardware range partitioner, or HARP, a streaming framework that offers a seamless execution environment for this and other streaming accelerators, and a detailed analysis of a 32nm physical design that matches the throughput of four to eight software threads while consuming just 6.9% of the area and 4.3% of the power of a Xeon core in the same technology generation.

Integrating frequent pattern clustering and branch-and-bound approaches for data partitioning

“…Moreover, for objective comparison, we examine five cases of transaction profiles by varying the numbers of independent queries and required attributes, as well as the frequencies, scan keys, selectivity, and lengths of attributes, as shown in Tables 11-15. Length of attributes = (8,8,8,8,4,8,8,12,20,22,4,8,6,5,3,30,12,8,6,6) …”

“…Their algorithm uses a large-neighborhood search technique with an analytical cost model to minimize the number of distributed transactions while controlling the amount of skew. Wu et al [30] not only proposed several software implementations of data partitioning but also deployed specialized hardware to further improve the efficiency of a parallel database. Fresno et al [10] presented an approach for integrating dense and sparse data management in parallel programming.…”

Compositional Dataflow Circuits

“…Conveyor. This network performs range partitioning [45]. The design chains n splitters (Figure 13) to partition an input stream into 2n + 1 output streams (e.g., 10 splitters yield a 21way Conveyor design).…”

Databases and hardware

“…Another approach is to use reconfigurable chips, such as field programmable gate arrays (FPGAs), to offload selected data processing paths. However, further demand for energy efficiency has sparked a number of proposals ranging from custom chips that accelerate individual operations such as partitioning and hashing, to query task accelerators, to designs that specialize chips to complete analytical queries [19].…”