Bita: Coverage-guided, automatic testing of actor programs

Tasharofi, Samira; Pradel, Michael; Lin, Yu Feng; Johnson, Ralph E.

doi:10.1109/ase.2013.6693072

Cited by 22 publications

(29 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Typically, these tools use a controlled scheduler to enforce a particular order of program events and reduce the state space by applying POR techniques or bounding the set of explored executions [Desai et al 2015]. With a goal of detecting bugs due to the ordering of two events, Bita [Tasharofi et al 2013] covers only the pairwise orderings of the program events. Recent work [Leesatapornwongsa et al 2014;Liu et al 2017;Lukman et al 2019] reduces the state space by using some semantic information about the system under test.…”

Section: Related Workmentioning

confidence: 99%

Trace aware random testing for distributed systems

Ozkan

Majumdar

Oraee

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Distributed and concurrent applications often have subtle bugs that only get exposed under specific schedules. While these schedules may be found by systematic model checking techniques, in practice, model checkers do not scale to large systems. On the other hand, naive random exploration techniques often require a very large number of runs to find the specific interactions needed to expose a bug. In recent years, several random testing algorithms have been proposed that, on the one hand, exploit state-space reduction strategies from model checking and, on the other, provide guarantees on the probability of hitting bugs of certain kinds. These existing techniques exploit two orthogonal strategies to reduce the state space: partial-order reduction and bug depth. Testing algorithms based on partial order techniques, such as RAPOS or POS, ensure nonredundant exploration of independent interleavings among system events by imposing an equivalence relation on schedules and ideally exploring only one schedule from each equivalence class. Techniques based on bug depth, such as PCT, exploit the empirical observation that many bugs are exposed by the clever scheduling of a small number of key events. They bias the sample space of schedules to only cover all executions of small depth, rather than the much larger space of all schedules. At this point, there is no random testing algorithm that combines the power of both approaches. In this paper, we provide such an algorithm. Our algorithm, trace-aware PCT (taPCT), extends and unifies several different algorithms in the random testing literature. It samples the space of low-depth executions by constructing a schedule online, while taking dependencies among events into account. Moreover, the algorithm comes with a theoretical guarantee on the probability of sampling a trace of low depthÐthe probability grows exponentially with the depth but only polynomially with the number of racy events explored. We further show that the guarantee is optimal among a large class of techniques. We empirically compare our algorithm with several state-of-the-art random testing approaches for concurrent software on two large-scale distributed systems, Zookeeper and Cassandra, and show that our approach is effective in uncovering subtle bugs and usually outperforms related random testing algorithms. CCS Concepts: • Mathematics of computing → Combinatorics; • Software and its engineering → Software testing and debugging; • Theory of computation → Generating random combinatorial structures.

show abstract

Section: Related Workmentioning

confidence: 99%

Trace aware random testing for distributed systems

Ozkan

Majumdar

Oraee

2019

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

show abstract

“…Message order violations have been collected under many different names in literature: data races by Petrov et al [44], harmful races by Raychev et al [46], order violations by Hong et al [31], and message ordering bugs by Tasharofi et al [55]. We consider message order violations to be a descriptive name while avoiding confusion with low-level data races present in thread-based programs.…”

Section: Comparison With Existing Terminology In Actor Literaturementioning

confidence: 99%

“…Tasharofi et al [55] identified twelve bugs in five Scala projects using the Akka actor library, which we categorize as message ordering problems. Bug-13 gives details of one of these bugs.…”

Section: Message Protocol Violationsmentioning

confidence: 99%

A Study of Concurrency Bugs and Advanced Development Support for Actor-based Programs

Lopez

Marr

Boix

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The actor model is an attractive foundation for developing concurrent applications because actors are isolated concurrent entities that communicate through asynchronous messages and do not share state. Thereby, they avoid concurrency bugs such as data races, but are not immune to concurrency bugs in general. This study taxonomizes concurrency bugs in actor-based programs reported in literature. Furthermore, it analyzes the bugs to identify the patterns causing them as well as their observable behavior. Based on this taxonomy, we further analyze the literature and find that current approaches to static analysis and testing focus on communication deadlocks and message protocol violations. However, they do not provide solutions to identify livelocks and behavioral deadlocks. The insights obtained in this study can be used to improve debugging support for actor-based programs with new debugging techniques to identify the root cause of complex concurrency bugs. 1 play() -> 2 Ping = spawn(fun ping/0), 3 spawn(fun() -> pong(Ping) end). 4 5 ping() -> 6 receive 7 pong_msg -> ok 8 end. 9 10 pong(Ping) -> 11 Ping ! pong_msg, 12 receive 13 ping_msg -> ok 14 end.Listing 1.1. Communication deadlock example in Erlang (from [15]). Line 12 has a blocking receive causing the pong process to deadlock because the expected message is never sent.

show abstract

“…Another line of research to help detecting concurrency bugs are approaches to influence the scheduler by adding random delays [15], by exploring interleavings systematically [36,12] or randomly [7], by forcing schedules that expose potential concurrency bugs [45,40,32,30,26], and by forcing schedules that cover yet uncovered interleavings [25,57,49]. By influencing the scheduler, these approaches also influence the performance of a concurrent program, and therefore are not directly applicable in our approach.…”

Section: Concurrency Bugsmentioning

confidence: 99%

“…To achieve correctness, developers can rely on bug finding techniques [17,16,6,46,26,41] and approaches that influence the schedule of concurrent executions [15,36,12,7,45,40,32,30,25,57,49]. In contrast, developers currently have only little support to measure, improve, and maintain the performance of thread-safe classes.…”

Section: Introductionmentioning

confidence: 99%

Performance regression testing of concurrent classes

Pradel

Huggler

Groß

2014

Proceedings of the 2014 International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

Developers of thread-safe classes struggle with two opposing goals. The class must be correct, which requires synchronizing concurrent accesses, and the class should provide reasonable performance, which is difficult to realize in the presence of unnecessary synchronization. Validating the performance of a thread-safe class is challenging because it requires diverse workloads that use the class, because existing performance analysis techniques focus on individual bottleneck methods, and because reliably measuring the performance of concurrent executions is difficult. This paper presents SpeedGun, an automatic performance regression testing technique for thread-safe classes. The key idea is to generate multi-threaded performance tests and to compare two versions of a class with each other. The analysis notifies developers when changing a thread-safe class significantly influences the performance of clients of this class. An evaluation with 113 pairs of classes from popular Java projects shows that the analysis effectively identifies 13 performance differences, including performance regressions that the respective developers were not aware of.

show abstract

Bita: Coverage-guided, automatic testing of actor programs

Cited by 22 publications

References 43 publications

Trace aware random testing for distributed systems

Trace aware random testing for distributed systems

A Study of Concurrency Bugs and Advanced Development Support for Actor-based Programs

Performance regression testing of concurrent classes

Contact Info

Product

Resources

About