Towards Testing ACID Compliance in the LDBC Social Network Benchmark

Abstract

Verifying ACID compliance is an essential part of database benchmarking, because the integrity of performance results can be undermined as the performance benefits of operating with weaker safety guarantees (at the potential cost of correctness) are well known. Traditionally, benchmarks have specified a number of tests to validate ACID compliance. However, these tests have been formulated in the context of relational database systems and SQL, whereas our scope of benchmarking are systems for graph data, many of which are non-relational. This paper presents a set of data model-agnostic ACID compliance tests for the LDBC (Linked Data Benchmark Council) Social Network Benchmark suite's Interactive (SNB-I) workload, a transaction processing benchmark for graph databases. We test all ACID properties with a particular emphasis on isolation, covering 10 transaction anomalies in total. We present results from implementing the test suite on 5 database systems.(LDBC) [3]. In particular, the LDBC's Social Network Benchmark Interactive workload (SNB-I) was designed to target transactional graph databases [10]. To provide protection against violations of correctness arising from the concurrent execution of transactions and system failures, such transactional databases provide Atomicity, Consistency, Isolation, and Durability (ACID) guarantees. Problem. Verifying ACID compliance is an important step in the benchmarking process for enabling fair comparison between systems. The performance benefits of operating with weaker safety guarantees are well established [13] but this can come at the cost of application correctness. To enable apples vs. apples performance comparisons between systems it is expected they uphold the ACID properties. Currently, LDBC provides no mechanism for validating ACID compliance within the SNB-I workflow. A simple solution would be to outsource the responsibility of demonstrating ACID compliance to benchmark implementors. However, the safety properties claimed by a system often do not match observable behaviour [14].To mitigate this problem, benchmarks such as TPC-C [20] include a number of ACID tests to be executed as part of the benchmarking auditing process. However, we found these tests cannot readily be applied to our context, as they assume lock-based concurrency control and an interactive query API that provides clients with explicit control over a transaction's lifecyle. Modern data systems often use optimistic concurrency control mechanisms [17] and offer a restricted query API, such as only executing transactions as stored procedures [19]. Further, tests that trigger and test row-level locking phenomena, for instance, do not readily map on graph database systems. Lastly, we found these tests are limited in the range of isolation anomalies they cover. Contribution. This paper presents the design of an implementation agnostic ACID compliance test suite for LDBC SNB-I 7 . Our guiding design principle was to be agnostic of system-level implementation details, relying solely on client observations ...