Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems

Pavlo, Andrew; Curino, Carlo; Zdonik, Stanley B.

doi:10.1145/2213836.2213844

Cited by 228 publications

(177 citation statements)

References 37 publications

(75 reference statements)

Supporting

Mentioning

173

Contrasting

Order By: Relevance

“…In [14], another automatic workload-aware database partitioning method is proposed along with an analytical model to estimate skew and coordination cost for DTs. It uses the same graph based workload representation of [5], and primarily focuses on optimal database design based on workload characteristics.…”

Section: Related Workmentioning

confidence: 99%

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Kamal

Murshed

Buyya

2016

Future Generation Computer Systems

View full text Add to dashboard Cite

h i g h l i g h t s• We propose incremental repartitioning of distributed OLTP databases for high-scalability.• We model two incremental repartitioning algorithm and lookup mechanism.• We develop a unique transaction generation model for simulation.• We derive novel impact metrics for distributed transactions.• Simulation results indicate adaptability of the methods scalable OLTP applications. On-line Transaction Processing (OLTP) applications often rely on shared-nothing distributed databases that can sustain rapid growth in data volume. Distributed transactions (DTs) that involve data tuples from multiple geo-distributed servers can adversely impact the performance of such databases, especially when the transactions are short-lived and these require immediate responses. The k-way min-cut graph clustering based database repartitioning algorithms can be used to reduce the number of DTs with acceptable level of load balancing. Web applications, where DT profile changes over time due to dynamically varying workload patterns, frequent database repartitioning is needed to keep up with the change. This paper addresses this emerging challenge by introducing incremental repartitioning. In each repartitioning cycle, DT profile is learnt online and k-way min-cut clustering algorithm is applied on a special sub-graph representing all DTs as well as those non-DTs that have at least one tuple in a DT. The latter ensures that the min-cut algorithm minimally reintroduces new DTs from the non-DTs while maximally transforming existing DTs into non-DTs in the new partitioning. Potential load imbalance risk is mitigated by applying the graph clustering algorithm on the finer logical partitions instead of the servers and relying on random one-to-one cluster-to-partition mapping that naturally balances out loads. Inter-server datamigration due to repartitioning is kept in check with two special mappings favouring the current partition of majority tuples in a cluster-the many-to-one version minimising data migrations alone and the one-toone version reducing data migration without affecting load balancing. A distributed data lookup process, inspired by the roaming protocol in mobile networks, is introduced to efficiently handle data migration without affecting scalability. The effectiveness of the proposed framework is evaluated on realistic TPC-C workloads comprehensively using graph, hypergraph, and compressed hypergraph representations used in the literature. To compare the performance of any incremental repartitioning framework without any bias of the external min-cut algorithm due to graph size variations, a transaction generation model is developed that can maintain a target number of unique transactions in any arbitrary observation window, irrespective of new transaction arrival rate. The overall impact of DTs at any instance is estimated from the exponential moving average of the recurrence period of unique transactions to avoid transient fluctuations. The effectiveness and adaptability of the proposed incremental repartition...

show abstract

Section: Related Workmentioning

confidence: 99%

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Kamal

Murshed

Buyya

2016

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…The workload consists of 5 queries, which are represented inside the fragments they access. There are 16 new data items, d 1 , ..., d 16 , that should be distributed over the fragments. The imbalance factor is ǫ s = 0.05, so resulting maximum size (taking into account new data items) is 42.…”

Section: Algorithmmentioning

confidence: 99%

“…In any case, it is below 5% for the worst case. To evaluate the quality of our partitioning approach, in addition to the partitioning efficiency metrics, as in [8,16] we studied the percentage of single-node queries, which means the percentage of the queries that can be executed by using the data of only one fragment. Figure 8 shows the results.…”

Section: Partitioning Efficiencymentioning

confidence: 99%

“…As a possible alternative they only consider hash-based partitioning over a single column. In [16], both hash and range-based partitioning on the most accessed attributes are considered for partitioning in OLTP systems. To find a near optimal solution, their approach explores a solution space by adapting the large-neighborhood search technique.…”

Section: Effect Of Imbalance Factor and Data Correlationmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Workload-Based Partitioning Algorithms for Continuously Growing Databases

Liroz-Gistau

Akbarinia

Pacitti

et al. 2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Applications with very large databases, where data items are continuously appended, are becoming more and more common. Thus, the development of efficient data partitioning is one of the main requirements to yield good performance. In the case of applications that have complex access patterns, e.g. scientific applications, workload-based partitioning could be exploited. However, existing workload-based approaches, which work in a static way, cannot be applied to very large databases. In this paper, we propose DynPart and DynPartGroup, two dynamic partitioning algorithms for continuously growing databases. These algorithms efficiently adapt the data partitioning to the arrival of new data elements by taking into account the affinity of new data with queries and fragments. In contrast to existing static approaches, our approach offers constant execution time, no matter the size of the database, while obtaining very good partitioning efficiency. We validated our solution through experimentation over real-world data; the results show its effectiveness.

show abstract

“…We place particular emphasis on the impact of the percentage of multipartition transactions. The percentage of multipartition transactions depends both on the workload properties and on the quality of the partitioning scheme, however, finding a good partitioning scheme for complex workloads remains an open problem [16,49,66]. In conjunction with the granularity of instances in a sharednothing deployment, the percentage of multipartition transactions determines the ratio of distributed transactions.…”

Section: Introductionmentioning

confidence: 99%

Characterization of the Impact of Hardware Islands on OLTP

Porobic

Pandis

Branco³

et al. 2015

The VLDB Journal

View full text Add to dashboard Cite

Modern hardware is abundantly parallel and increasingly heterogeneous. The numerous processing cores have non-uniform access latencies to the main memory and processor caches, which causes variability in the communication costs. Unfortunately, database systems mostly assume that all processing cores are the same and that microarchitecture differences are not significant enough to appear in critical database execution paths. As we demonstrate in this paper, however, non-uniform core topology does appear in the critical path and conventional database architectures achieve suboptimal and even worse, unpredictable performance. We perform a detailed performance analysis of OLTP deployments in servers with multiple cores per CPU (multicore) and multiple CPUs per server (multisocket). We compare different database deployment strategies where we vary the number and size of independent database instances running on a single server, from a single shared-everything instance to fine-grained shared-nothing configurations. We quantify the impact of non-uniform hardware on various deployments by (a) examining how efficiently each deployment uses the available hardware resources and (b) measuring the impact of distributed transactions and skewed requests on different workloads. We show that no strategy is optimal for all cases and that the best choice depends on the combination of hardware topology and workload characteristics. Finally, we argue that transaction processing systems must be aware of the hardware topology in order to achieve predictably high performance.

show abstract

Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems

Cited by 228 publications

References 37 publications

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Workload-aware incremental repartitioning of shared-nothing distributed databases for scalable OLTP applications

Dynamic Workload-Based Partitioning Algorithms for Continuously Growing Databases

Characterization of the Impact of Hardware Islands on OLTP

Contact Info

Product

Resources

About