Automated Detection of Test Fixture Strategies and Smells

Abstract

Designing automated tests is a challenging task. One important concern is how to design test fixtures, i.e. code that initializes and configures the system under test so that it is in an appropriate state for running particular automated tests. Test designers may have to choose between writing in-line fixture code for each test or refactor fixture code so that it can be reused for other tests. Deciding on which approach to use is a balancing act, often trading off maintenance overhead with slow test execution. Additionally, over time, test code quality can erode and test smells can develop, such as the occurrence of overly general fixtures, obscure in-line code and dead fields. In this paper, we show that test smells related to fixture set-up occur in industrial projects. We present a static analysis technique to identify fixture related test smells. We implemented this test analysis technique in a tool, called TestHound, which provides reports on test smells and recommendations for refactoring the smelly test code. We evaluate the tool through three industrial case studies and show that developers find that the tool helps them to understand, reflect on and adjust test code.

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

“…Unlike other existing detection tools (e.g., [35] or [36]), this tool can identify all the test smells considered in this study. Basically, the tool relies on the detection rules described in Table II.…”

“…Later on, Meszaros defined other smells affecting test code [29]. Based on these catalogs, Greiler et al [35], [62] showed that test smells affecting test fixtures frequently occur in industrial contexts, and for this reason they presented TESTHOUND, a tool able to identify fixturerelated test smells such as General Fixture or Vague Header Setup [35]. Van Rompaey et al [36] devised a heuristic code metric-based technique able to identify instances of two test smells, i.e., General Fixture and Eager Test, but the results of an empirical study showed the low accuracy of the approach.…”

Strategies for avoiding text fixture smells during software evolution

“…Otherwise, they end up with poor solutions to recurring implementation and design problems in their test code, also known as test smells [6]. To support developers during the analysis and adjustment of test fixtures, we previously developed a tool called TestHound 2 to automatically detect test fixture smells and guide test code refactoring [7].…”

On the effectiveness of unit tests in test-driven development

“…Garousi et al [43] highlights that around half of primary studies investigated test smells, whereas other studies used coverage as an indicator of test code/ assertions quality. Some studies on test smells argue that these issues degrade the quality of test code since test smells prevent the maintainability of test codes written by the development teams [20,35]. Madeyski [26] on the other hand discusses the quality of test suite in terms of their thoroughness and effectiveness: Coverage metrics are indicators of test suite thoroughness, while mutation score indicator is found to be effective at finding faults [26].…”

“…Studies on test smells often propose new test smell sub-categories such as general fixture and eager, and methods to detect those (e.g. [20], [35]), or they study the impact of test smells on maintenance activities (e.g. [32]).…”