ACaZoo: A Distributed Key-Value Store Based on Replicated LSM-Trees

Abstract

In this paper we describe the design and implementation of ACaZoo 1 , a key-value store that combines strong consistency with high performance and high availability. ACaZoo supports the popular column-oriented data model of Apache Cassandra and HBase. It implements strongly-consistent data replication using primary-backup atomic broadcast of a writeahead log, which records data mutations to a Log-structured Merge Tree (LSM-Tree). ACaZoo scales by horizontally partitioning the key space via consistent primary-key hashing on available replica groups (RGs). LSM-Tree compactions can hamper performance, especially when they take place at RG primaries. ACaZoo addresses this problem by changing RG leadership prior to heavy compactions, a method that can improve throughput by up to 40% in write-intensive workloads. We evaluate ACaZoo using the Yahoo Cloud Serving Benchmark (YCSB) and compare it to Oracle's NoSQL Database and to Cassandra providing serial consistency via an extension of the Paxos algorithm.

Addressing the Read-Performance Impact of Reconfigurations in Replicated Key-Value Stores

“…Production systems at Google [20] trigger a reconfiguration as often as every 30 minutes when the system re-evaluates the optimal placement of a leader or its replicas based on workload characteristics, or every 2 hours in the case of Microsoft's store [18] that adapts to the shift of traffic across different time zones. Frequent reconfiguration actions may also be used as a lightweight adaptation mechanism to hide internal performance bottlenecks or in the case of colocated resource intensive background tasks on the serving replicas [22], [23]. In all these adaptation mechanisms, the newly assigned read-serving replicas (although consistent at the level of persisted data) may have missed recent reads and thus have an outdated memory cache leading to a performance impact (Section 4).…”

“…In all these adaptation mechanisms, the newly assigned read-serving replicas (although consistent at the level of persisted data) may have missed recent reads and thus have an outdated memory cache leading to a performance impact (Section 4). Empirical evidence of the challenge addressed in this paper (and inspiration for this work) is provided by our own previous research [22], [23], [32]. Figure 1 (from [22]) demonstrates that the action of changing the primary replica can hide the performance impact of a backup task, however the improved system (Figure 1b) still suffers from a smaller but non-negligible performance hit (area in red circle) due to cold-cache misses at the new primary (a previously non-read-serving replica).…”

“…We thus moved towards scenarios where we switch the primary node within the replica group. This may occur during a failover action or to improve performance [18], [19], [22], [23]. In this case, all transactions are executed in the node that serves as primary.…”

A Queueing Network Model for Performance Prediction of Apache Cassandra

“…To achieve higher performance, NoSQL databases prefer eventually consistency instead of strong consistency. For this reason, to achieve strong consistency and meet the consistency levels required by the application without sacrificing the performance, variations of Cassandra database have been developed [8,14]. Chihoub et al presents a self-tuning mechanism to change at run-time query consistency levels, taking in consideration system state.…”

“…Chihoub et al presents a self-tuning mechanism to change at run-time query consistency levels, taking in consideration system state. Differently, in [14] the Cassandra's node architecture is modified to always achieve strong consistency of the data.…”