Ballerina: Automatic generation and clustering of efficient random unit tests for multithreaded code

Abstract

Abstract-Testing multithreaded code is hard and expensive. A multithreaded unit test creates two or more threads, each executing one or more methods on shared objects of the class under test. Such unit tests can be generated at random, but basic random generation produces tests that are either slow or do not trigger concurrency bugs. Worse, such tests have many false alarms, which require human effort to filter out.We present BALLERINA, a novel technique for automated random generation of efficient multithreaded tests that effectively trigger concurrency bugs. BALLERINA makes tests efficient by having only two threads, each executing a single, randomly selected method. BALLERINA increases chances that such simple parallel code finds bugs by appending it to more complex, randomly generated sequential code. We also propose a clustering technique to reduce the manual effort in inspecting failures of automatically generated multithreaded tests. We evaluate BALLERINA on 14 real-world bugs from six popular codebases: Groovy, JDK, JFreeChart, Apache Log4j, Apache Lucene, and Apache Pool. The experiments show that tests generated by BALLERINA find bugs on average 2X-10X faster than basic random generation, and our clustering technique reduces the number of inspected failures on average 4X-8X. Using BALLERINA, we found three previously unknown bugs, two of which were already confirmed and fixed.

Automatic patch generation learned from human-written patches

“…For our experiment, six open source projects including Mozilla Rhino, Eclipse AspectJ, Apache Log4j, and Apache Commons (Math, Lang, Collections) were selected, since they are commonly used in the literature [6], [19], [20] and have well-maintained bug reports. We tried to search their corresponding issue trackers for reproducible bugs.…”

Generating Unit Tests for Concurrent Classes

“…Ballerina [19] uses random test prefixes, and then explores two methods executed in parallel in two different threads. Sen and Agha [22] combined jCute [23] with race detection algorithms.…”

“…Sen and Agha [22] combined jCute [23] with race detection algorithms. The Ballerina tool [19] is the most similar to the approach described in this paper: It randomly invokes methods on a shared object under test in parallel. However, it cannot find concurrency issues that require execution of parallel sequences of code rather than individual method calls, and it cannot find data races based on more than two memory access points [16] (e.g., Figure 4).…”

“…If two threads access one instance of the FastTreeMap class, one calls setFast(true) once and after that always calls get('A') until a value for 'A' is found, while the other calls put('A', 'B') once, the compiler (which in the Java memory model also includes the memory hierarchy) is free to never return a value different from NULL to the first thread. Because of errors like this, it is not enough to check for linearizability [19]. The compiler has some degrees of freedom and may choose to only display this error on a production system, e.g., if the production system has more processor caches.…”

Automatic testing of sequential and concurrent substitutability

“…First, a recent study on real-world concurrency bugs shows that most bugs (96%) can be reproduced with two threads [11]. Second, the execution of concurrent suffixes with a single method call can be efficiently explored and still reveals many real-world bugs [12].…”