Automated Data Partitioning for Highly Scalable and Strongly Consistent Transactions

Abstract

Modern transactional processing systems need to be fast and scalable, but this means many such systems settled for weak consistency models. It is however possible to achieve all of strong consistency, high scalability and high performance, by using fine-grained partitions and light-weight concurrency control that avoids superfluous synchronization and other overheads such as lock management. Independent transactions are one such mechanism, that rely on good partitions and appropriately defined transactions. On the downside, it is not usually straightforward to determine optimal partitioning schemes, especially when dealing with non-trivial amounts of data. Our work attempts to solve this problem by automating the partitioning process, choosing the correct transactional primitive, and routing transactions appropriately.

Helenos: A realistic benchmark for distributed transactional memory

“…Among them, there is an immensely popular series of TPC benchmarks. As shown in the work of Turcu et al, they can be altered to work as an evaluation tool for distributed TMs, but they have their own share of drawbacks.…”

A Replication Scheme for Multiple Fragmentations with Overlapping Fragments

“…That approach was later implemented in practice in the "Relational Cloud" database system [12]. More recently in a competitor system, [13] reduce the amount of inter-partition dependencies and implement an advanced transaction routing.…”

“…The variable x l ik represents the placement of the lth copy of fragment f i on server k. Equation (13) demands that each of the m copies of each fragment is assigned to one server. Equation 14assigns to each copy of fragment f i the capacity consumption w i and ensures that the maximum capacity of each server is not exceeded.…”

STI-BT: A Scalable Transactional Index

“…Consistent hashing is particularly attractive in large scale, elastic stores, as it avoids reliance on external lookup services, and allows to minimize the keys to be redistributed upon changes of the system's scale. However, deterministic random hash functions to map keys identifiers to machines lead to poor data locality [25], [26]. Hence, DKV stores using consistent hashing (like Infinispan) typically allow programmers to provide custom functions.…”