Workload-aware data partitioning in community-driven data grids

Abstract

Collaborative research in various scientific disciplines requires support for scalable data management enabling the efficient correlation of globally distributed data sources. Motivated by the expected data rates of upcoming projects and a growing number of users, communities explore new data management techniques for achieving high throughput. Community-driven data grids deliver such highthroughput data distribution for scientific federations by partitioning data according to application-specific data and query characteristics. Query hot spots are an important and challenging problem in this environment. Existing approaches to load-balancing from Peer-to-Peer (P2P) data management and sensor networks do not directly meet the requirements of a data-intensive e-science environment. In this paper, our contributions are partitioning schemes based on multi-dimensional index structures enabling communities to trade off data load balancing and handling query hot spots via splitting and replication. We evaluate the partitioning schemes with two typical kinds of data sets from the astrophysics domain and workloads extracted from Sloan Digital Sky Survey (SDSS) query traces and perform throughput measurements in real and simulated networks. The experiments demonstrate the improved workload distribution capabilities and give promising directions for the development of future community grids.

Partout: A Distributed Engine for Efficient RDF Processing

“…Further systems applying replication have been proposed on top of P2P architectures. Still, for our problem scenario we cannot make use of them because they either consider each triple separately [12] or make assumptions on queries [30] that do not conform to the general-purpose setup we are considering in this paper.…”

Holistic Load-Balancing in a Distributed Spatial Cache

“…Others, e.g. [4], address only static workload imbalances which are well-known before execution. Examples for such imbalances are data access patterns which follow Zipf's law or which relate to the population density, as users typically access data concerning their surroundings.…”

An Elastic Cache Infrastructure through Multi-level Load-balancing

“…Existing load-balancing mechanisms are limited to one of the three above-mentioned indicators. For instance, (Aberer et al, 2005), focuses on data skew only, (Scholl et al, 2009), addresses only expected workload and many other approaches, such as (Lübbe et al, 2012;Wang et al, 2005), solely focus on dynamic load-peaks. Fully relying on a single indicator can lead to inadequate resource allocation during load-balancing.…”