Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities

Panichella, Annibale; Panichella, Sebastiano; Sawant, Anand Ashok; Hellendoorn, Vincent J.

doi:10.1109/icsme46990.2020.00056

Cited by 21 publications

(12 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These approaches rely on detection rules that raise warnings when some metrics (e.g., size, number of method calls, number of assertions) in the test code exceed given thresholds. However, a recent study [24] showed that detection rules are far less accurate than previously reported, especially when tests use external resources.…”

Section: Related Workmentioning

confidence: 80%

Log-based slicing for system-level test cases

Messaoudi

Shin

Panichella

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

Regression testing is arguably one of the most important activities in software testing. However, its cost-effectiveness and usefulness can be largely impaired by complex system test cases that are poorly designed (e.g., test cases containing multiple test scenarios combined into a single test case) and that require a large amount of time and resources to run. One way to mitigate this issue is decomposing such system test cases into smaller, separate test cases-each of them with only one test scenario and with its corresponding assertions-so that the execution time of the decomposed test cases is lower than the original test cases, while the test effectiveness of the original test cases is preserved. This decomposition can be achieved with program slicing techniques, since test cases are software programs too. However, existing static and dynamic slicing techniques exhibit limitations when (1) the test cases use external resources, (2) code instrumentation is not a viable option, and (3) test execution is expensive.In this paper, we propose a novel approach, called DS3 (Decomposing System teSt caSe), which automatically decomposes a complex system test case into separate test case slices. The idea is to use test case execution logs, obtained from past regression testing sessions, to identify "hidden" dependencies in the slices generated by static slicing. Since logs include run-time information about the system under test, we can use them to extract access and usage of global resources and refine the slices generated by static slicing.We evaluated DS3 in terms of slicing effectiveness and compared it with a vanilla static slicing tool. We also compared the slices obtained by DS3 with the corresponding original system test cases, in terms of test efficiency and effectiveness. The evaluation results on one proprietary system and one open-source system show that DS3 is able to accurately identify the dependencies related to the usage of global resources, which vanilla static slicing misses. Moreover, the generated test case slices are, on average, 3.56 times faster than original system test cases and they exhibit no significant loss in terms of fault detection effectiveness. CCS CONCEPTS• Software and its engineering → Software testing and debugging; Automated static analysis.

show abstract

Section: Related Workmentioning

confidence: 80%

Log-based slicing for system-level test cases

Messaoudi

Shin

Panichella

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

show abstract

“…While our objective here is not to discredit the validity of the current set of test smell detection tools, we only highlight inconsistencies that might lead to research studies obtaining varying results based on the tool in use. As stated by Panichella et al [61], there is a need for a community-maintained gold-set/standard of smelly test files to validate the current and future smell detection tools. We understand that the process involved in the creation of a community-curated gold-set might be time-consuming.…”

Section: Discussionmentioning

confidence: 99%

Test Smell Detection Tools: A Systematic Mapping Study

Aljedaani¹,

Peruma²,

Aljohani³

et al. 2021

Preprint

View full text Add to dashboard Cite

Test smells are defined as sub-optimal design choices developers make when implementing test cases. Hence, similar to code smells, the research community has produced numerous test smell detection tools to investigate the impact of test smells on the quality and maintenance of test suites. However, little is known about the characteristics, type of smells, target language, and availability of these published tools. In this paper, we provide a detailed catalog of all known, peer-reviewed, test smell detection tools.We start with performing a comprehensive search of peer-reviewed scientific publications to construct a catalog of 22 tools. Then, we perform a comparative analysis to identify the smell types detected by each tool and other salient features that include programming language, testing framework support, detection strategy, and adoption, among others. From our findings, we discover tools that detect test smells in Java, Scala, Smalltalk, and C++ test suites, with Java support favored by most tools. These tools are available as command-line and IDE plugins, among others. Our analysis also shows that most tools overlap in detecting specific smell types, such as General Fixture. Further, we encounter four types of techniques these tools utilize to detect smells. We envision our study as a onestop source for researchers and practitioners in determining the tool appropriate for their needs. Our findings also empower the community with information to guide future tool development.

show abstract

“…To improve the readability of the generated tests and to avoid test smells [8], EVOSUITE applies post-processing once the available search budget has been exhausted [2,3]. In particular, minimisation is used to remove redundant tests and statements, and a minimised set of assertions is added using mutation analysis [6].…”

Section: About Evosuitementioning

confidence: 99%