Trace aware random testing for distributed systems

Abstract

Distributed and concurrent applications often have subtle bugs that only get exposed under specific schedules. While these schedules may be found by systematic model checking techniques, in practice, model checkers do not scale to large systems. On the other hand, naive random exploration techniques often require a very large number of runs to find the specific interactions needed to expose a bug. In recent years, several random testing algorithms have been proposed that, on the one hand, exploit state-space reduction strategies from model checking and, on the other, provide guarantees on the probability of hitting bugs of certain kinds. These existing techniques exploit two orthogonal strategies to reduce the state space: partial-order reduction and bug depth. Testing algorithms based on partial order techniques, such as RAPOS or POS, ensure nonredundant exploration of independent interleavings among system events by imposing an equivalence relation on schedules and ideally exploring only one schedule from each equivalence class. Techniques based on bug depth, such as PCT, exploit the empirical observation that many bugs are exposed by the clever scheduling of a small number of key events. They bias the sample space of schedules to only cover all executions of small depth, rather than the much larger space of all schedules. At this point, there is no random testing algorithm that combines the power of both approaches. In this paper, we provide such an algorithm. Our algorithm, trace-aware PCT (taPCT), extends and unifies several different algorithms in the random testing literature. It samples the space of low-depth executions by constructing a schedule online, while taking dependencies among events into account. Moreover, the algorithm comes with a theoretical guarantee on the probability of sampling a trace of low depthÐthe probability grows exponentially with the depth but only polynomially with the number of racy events explored. We further show that the guarantee is optimal among a large class of techniques. We empirically compare our algorithm with several state-of-the-art random testing approaches for concurrent software on two large-scale distributed systems, Zookeeper and Cassandra, and show that our approach is effective in uncovering subtle bugs and usually outperforms related random testing algorithms. CCS Concepts: • Mathematics of computing → Combinatorics; • Software and its engineering → Software testing and debugging; • Theory of computation → Generating random combinatorial structures.

Model‐checking‐driven explorative testing of CRDT designs and implementations

“…Instead of systematically testing the distributed system, randomized concurrency testing tackles the intractable test space via sampling. Probabilistic concurrency testing (PCT) 57,58 and partial order sampling (POS) 59 are two sampling strategies used to test distributed systems these years. They provide probabilistic guarantees to find concurrency bugs with certain patterns at a reasonable cost.…”

Automated Test Case Generation for Service Composition from Event Logs

“…A plethora of works have been proposed to generate test cases without specification, by using random testing or model learning. A few of them are specialised to communicating systems (Ozkan et al, 2019;Arcuri, 2018;Petrenko and Avellaneda, 2019;Aarts et al, 2014;Tian et al, 2017). For instance, Arcuri proposed algorithms to create test suites for Web service compositions (Arcuri, 2018) by considering the test case generation as a a multi-objective problem, whose objectives are related to metrics over source code properties.…”

Testing consensus implementations using communication closure

“…In our evaluation, none of those tests could hit the bug in the system. The infrequency of hitting the bug is not surprising since the bug in Cassandra is known to be a difficult bug and it is reproduced only in few executions in previous works [Leesatapornwongsa et al 2014;Ozkan et al 2019].…”

“…Using algorithmic insights into testing distributed systems to reduce the space of executions is a point of departure from existing work in randomized or systematic testing of implementations of distributed systems. At the same time, our insight is orthogonal to the many reduction techniques already exploited in existing tools, such as depth bounding [Ozkan et al 2018], partial order reduction [Ozkan et al 2019;Yuan et al 2018], or semantics-aware analyses [Leesatapornwongsa et al 2014;Lukman et al 2019].…”

“…Testing implementations of distributed systems has received considerable attention over the recent years. Probabilistic Concurrency Testing with Chain Partitioning (PCTCP) [24,25] provides precise probabilistic guarantees about observing every possible order between a fixed number of events. The Jepsen tool [1] makes it possible to introduce benign faults randomly with a certain frequency, which provides very little control over the outcome of a test.…”