Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation

Abstract

Automated test case generation is an effective technique to yield high-coverage test suites. While the majority of research effort has been devoted to satisfying coverage criteria, a recent trend emerged towards optimizing other non-coverage aspects. In this regard, runtime and memory usage are two essential dimensions: less expensive tests reduce the resource demands for the generation process and for later regression testing phases. This study shows that performance-aware test case generation requires solving two main challenges: providing accurate measurements of resource usage with minimal overhead and avoiding detrimental effects on both final coverage and fault detection effectiveness. To tackle these challenges we conceived a set of performance proxies (inspired by previous work on performance testing) that provide an approximation of the test execution costs (i.e., runtime and memory usage). Thus, we propose an adaptive strategy, called pDynaMOSA, which leverages these proxies by extending DynaMOSA, a state-of-the-art evolutionary algorithm in unit testing. Our empirical study -involving 110 non-trivial Java classes-reveals that our adaptive approach has comparable results to DynaMOSA over seven different coverage criteria (including branch, line, and weak mutation coverage) and similar fault detection effectiveness (measured via strong mutation coverage). Additionally, we observe statistically significant improvements regarding runtime and memory usage for test suites with a similar level of target coverage. Our quantitative and qualitative analyses highlight that our adaptive approach facilitates selecting better test inputs, which is an essential factor to test production code with fewer resources.

Java unit testing tool competition

“…Future directions for future unit testing tool competitions go toward (i) the comparison of tools by considering additional criteria than the coverage and mutation analysis [5]; (ii) exploring the possibility to consider other languages (e.g., Python) in further competitions; (iii) extend even further the dockerized version of the infrastructure, making it available as service to researchers of the community.…”

Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor

“…Finally, we plan to investigate the use of SDC-Scissor in other CPS domains, such as drones, to investigate how it performs when testing focuses on different types of safety-critical faults. Specifically, it is important to investigate approaches that are more human-oriented or are able to integrate humans into-the-loop [36,37], via multi-objective optimizations [41,42].…”

Selectively Combining Multiple Coverage Goals in Search-Based Unit Test Generation

“…Panichella et al [38,39] introduced many-objective optimization algorithms. Grano et al [21] proposed a variant of DynaMOSA [39] to reduce the computation costs; (2) fitness gradients recovery: Lin et al [31] proposed an approach to address the inter-procedural flag problem. Lin et al [30] proposed a test seed synthesis approach to create complex test inputs.…”