Concurrency Testing Using Controlled Schedulers

Abstract

We present an independent empirical study on concurrency testing using controlled schedulers. We have gathered 49 buggy concurrent software benchmarks, drawn from public code bases, which we call SCTBench. We applied a modified version of an existing concurrency testing tool to SCTBench, testing five controlled scheduling techniques: depth-first search, preemption bounding, delay bounding, a controlled random scheduler, and probabilistic concurrency testing (PCT). We attempt to answer several research questions, including: Which technique performs the best, in terms of bug finding ability? How effective are the two main schedule bounding techniques, preemption bounding and delay bounding, at finding bugs? What challenges are associated with applying concurrency testing techniques to existing code? Can we classify certain benchmarks as trivial or non-trivial? Overall, we found that PCT (with parameter d=3) was the most effective technique in terms of bug finding; it found all the bugs found by the other techniques, plus an additional three, and it missed only one bug. Surprisingly, we found that the "naïve" controlled random scheduler, which randomly chooses one thread to execute at each scheduling point, performs well, finding more bugs than preemption bounding and just two fewer bugs than delay bounding. Our findings confirm that delay bounding is superior to preemption bounding and schedule bounding is superior to an unbounded depth-first search. The majority of bugs in SCTBench can be exposed using a small schedule bound (1-2), supporting previous claims, although one benchmark requires 5 preemptions. We found that the need to remove nondeterminism and control all synchronization (as is required for systematic concurrency testing) can be nontrivial. There were 8 distinct programs that could not easily be included in out study, such as those that perform network and inter-process communication. We report various properties about the benchmarks tested, such as the fact that the bugs in 18 benchmarks were exposed 50% of the time when using random scheduling. We note that future work should not use the benchmarks that we classify as trivial when presenting new techniques, other than as a minimum baseline. We have made SCTBench and our tools publicly available for reproducibility and use in future work.

Optimal stateless model checking for reads-from equivalence under sequential consistency

“…Thus an eager technique would explore a constant number of traces and spend time linear in the parameter n, whereas a lazy technique would explore a number of traces, and spend time exponential in n. □ Benchmarks and Platform. The benchmark programs we use in this section all are parametric in the number of threads, and are taken from SV-COMP [2019], SCTBench [Thomson et al 2016], and from the papers that describe the tools we use [Abdulla et al 2017;Aronis et al 2018;Chalupa et al 2018]. In order for these programs to be handled by all tools, and RCMC in particular, we needed to convert some of them to C11 with SC read and write accesses.…”

“…Starting from five programs of SCTBench [Thomson et al 2016], the Systematic Concurrency Testing Benchmark Suite, we conducted an experiment in order to see whether the rf equivalence provides some advantage as far as bug finding is concerned. For our experiment, we chose the five reorder_N_bad benchmarks (N ∈ {3,4,5,10,20} is the number of created pthreads).…”

“…These are programs adversarial for delay bounding; the smallest delay bound required for the bug to manifest is incremented as the thread count is incremented. Thomson et al [2016] report that, although many systematic concurrency testing techniques locate the bug for small thread counts (N ∈ {3,4,5}), only probabilistic concurrency testing with larger d values [Burckhardt et al 2010] manages to hit the bug on reorder_10_bad and reorder_20_bad. (This is on an experiment that runs each technique for 100 000 executions.)…”

Partial Order Aware Concurrency Sampling

“…The most straightforward sampling algorithm is random walk : at each step, randomly pick an enabled event to execute. Previous work showed that even such a sampling outperformed the exhaustive search at finding errors in real-world concurrent programs [24]. This can be explained by applying the small-scope hypothesis [12,Sect.…”

Testing and Verifying Chain Repair Methods for Corfu Using Stateless Model Checking

“…The justification is that common patterns of concurrency bugs require few scheduling constraints and these in turn can be related to few preemptions [3,18]. Delay bounding [6] is another bounding technique that forces the scheduler to always schedule the first non-blocked process out of a total order of all processes.…”