Investigating Severity Thresholds for Test Smells

Spadini, Davide; Schvarcbacher, Martin; Oprescu, Ana-Maria; Bruntink, Magiel; Bacchelli, Alberto

doi:10.1145/3379597.3387453

Cited by 41 publications

(29 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The notion of code smells was later extended to test code [1]. To help developers avoid such issues, several tools have been introduced to automatically flag smelly test code [2], [3]. This has in turn enabled researchers to empirically study the prevalence of test smells and their effects, confirming that test smells are not only common in open source and industrial software, but also have a strong negative impact on program comprehension and maintenance [2], [5], [10].…”

Section: A Test Smellsmentioning

confidence: 99%

“…For the post-processing steps, the EVOSUITE default settings use 60 seconds for test case minimization, and a further 60 seconds for assertions generation/minimization. These parameter values can be derived from the EVOSUITE code and from the replication package by Grano et al 2 If the test case minimization and the assertion generation reach their respective time-outs, EVOSUITE terminates the post-processing and returns the original, nonminimized, test suites. To ensure that this phase completes successfully, we thus increase the timeouts from 60 seconds (used in [6]) to ten minutes; this also leads to higher coverage.…”

Section: B Test Case Generationmentioning

confidence: 99%

“…The second detection tool we study is TSDETECT [10], which is publicly available on GitHub. 3 Recently, Spadini et al [2] calibrated the detection rules in TSDETECT based on developers' perception and classification of test smell severity, resulting in thresholds that are better aligned with what developers consider actual bad test design choices.…”

Section: Detection Tool Selectionmentioning

confidence: 99%

“…An ambitious goal of software engineering research is to automatically assess the quality and maintainability of test code. Automatically identifying this is challenging [1], [2], as the notion of quality is both context-sensitive and subject to change over time. A common approach is to focus on those parts of a program that clearly violate certain wellestablished rules informed by practice; e.g., that a single test method should not test multiple requirements [3], [4].…”

Section: Introductionmentioning

confidence: 99%

“…Recent work showed that test smells are especially prevalent in automatically generated test-cases [6]; however, the heuristics defined typically apply to the goal of detecting smells in manually written test cases, which may not be directly transferrable to automatically generated test suites, as those tests are often markedly different in structure and semantics from their human-written counterparts. Furthermore, while these rules have been designed based on the original definition of "test smells", a recent empirical study showed that developers do not perceive these as reflective of test suite quality and maintainability, finding them overly strict [2].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities

Panichella

Sawant

et al. 2020

2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

Test smells attempt to capture design issues in test code that reduce their maintainability. Previous work found such smells to be highly common in automatically generated test-cases, but based this result on specific static detection rules; although these are based on the original definition of "test smells", a recent empirical study showed that developers perceive these as overly strict and non-representative of the maintainability and quality of test suites. This leads us to investigate how effective such test smell detection tools are on automatically generated test suites. In this paper, we build a dataset of 2,340 test cases automatically generated by EVOSUITE for 100 Java classes. We performed a multi-stage, cross-validated manual analysis to identify six types of test smells and label their instances. We benchmark the performance of two test smell detection tools: one widely used in prior work, and one recently introduced with the express goal to match developer perceptions of test smells. Our results show that these test smell detection strategies poorly characterized the issues in automatically generated test suites; the older tool's detection strategies, especially, misclassified over 70% of test smells, both missing real instances (false negatives) and marking many smell-free tests as smelly (false positives). We identify common patterns in these tests that can be used to improve the tools, refine and update the definition of certain test smells, and highlight as of yet uncharacterized issues. Our findings suggest the need for (i) more appropriate metrics to match development practice; and (ii) more accurate detection strategies, to be evaluated primarily in industrial contexts.

show abstract

Section: A Test Smellsmentioning

confidence: 99%

Section: B Test Case Generationmentioning

confidence: 99%

Section: Detection Tool Selectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities

Panichella

Sawant

et al. 2020

2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

show abstract

On the diffusion of test smells and their relationship with test code quality of Java projects

Martins

Ferreira

Machado

2023

J Software Evolu Process

View full text Add to dashboard Cite

Test smells are considered bad practices that can reduce the test code quality, thus harming software testing goals and maintenance activities. Prior studies have investigated the diffusion of test smells and their impact on test code maintainability. However, we cannot directly compare the outcomes of the studies as most of them use customized datasets. In response, we introduced the TSSM (Test Smells and Structural Metrics) dataset, containing test smells detected using the JNose Test tool and structural metrics (test code and production code) calculated with the CK metrics tool of 13,703 open‐source Java systems from GitHub. In addition, we perform an empirical study to investigate the relationship between test smells and structural metrics of test code and the relationship between test smells on a large‐scale dataset. We split the projects into three clusters to analyze the distribution of test smells, the co‐occurrences among test smells, and the correlation of test smells and structural metrics of test code. The ratio of smelly test classes with a specific test smell is similar among the clusters, but we could observe a significant difference in the number of test smells among them. The test smells Sleepy Test, Mystery Guest, and Resource Optimism rarely occur in the three clusters, and the last two are strongly correlated, indicating that those test smells are more severe than others. Our results point out that most test smells have a moderate correlation with high complexity, large size, and coupling of the test code, indicating that they can also negatively affect its quality. To support further studies, we made our dataset publicly available.

show abstract