On the test smells detection: an empirical study on the JNose Test accuracy

Abstract

Several strategies have supported test quality measurement and analysis. For example, code coverage, a widely used one, enables verification of the test case to cover as many source code branches as possible. Another set of affordable strategies to evaluate the test code quality exists, such as test smells analysis. Test smells are poor design choices in test code implementation, and their occurrence might reduce the test suite quality. A practical and largescale test smells identification depends on automated tool support. Otherwise, test smells analysis could become a cost-ineffective strategy. In an earlier study, we proposed the JNose Test, automated tool support to detect test smells and analyze test suite quality from the test smells perspective. This study extends the previous one in two directions: i) we implemented the JNose-Core, an API encompassing the test smells detection rules. Through an extensible architecture, the tool is now capable of accomodating new detection rules or programming languages; and ii) we performed an empirical study to evaluate the JNose Test effectiveness and compare it against the state-of-the-art tool, the tsDetect. Results showed that the JNose-Core precision score ranges from 91% to 100%, and the recall score from 89% to 100%. It also presented a slight improvement in the test smells detection rules compared to the tsDetect for the test smells detection at the class level.

On the diffusion of test smells and their relationship with test code quality of Java projects

“…The tsDetect tool presents a precision score ranging from 85% to 100% and a recall score ranging from 90% to 100%. The JNose Test tool was built based on the tsDetect tool, making the execution process fully automated through an API (Application Programming Interface) and keeping a precision score ranging from 85% to 100% and a recall score from 90% to 100% 18,23 in coarse granularity (class level). In addition, the JNose Test tool detects test smells in fine granularity (method, block, and line levels) with a precision score ranging from 84% to 100% and a recall score from 47% to 100%.…”

“…The authors selected 47 peer‐reviewed papers published until December 2020. Their contributions include a list of 22 tools and comparing them regarding the supported test smells, testing frameworks, and detection strategies (e.g., metrics, 21,43 rules, 18,19,44,45 dynamic tainting, 46,47 and information retrieval 3,44 ). In summary, the tools support the detection of 66 test smells, and four tools support the refactoring of 10 test smells among seven different types of testing frameworks.…”

“…two key empirical studies: the first was an exploratory study on the diffusion of nine test smells in 18 software projects, and the second was a controlled experiment with 20 participants to investigate the effects of test smells on software maintenance. Later, Bavota et al 11 21,43 rules, 18,19,44,45 dynamic tainting, 46,47 and information retrieval 3,44 ). In summary, the tools support the detection of 66 test smells, and four tools support the refactoring of 10 test smells among seven different types of testing frameworks.…”

“…Some studies have investigated whether the diffusion of test smells could impact the test code maintainability 11,15 and whether test smells could co‐occur together or with code properties (number of classes and lines) 5,16,17 . Other studies have investigated the developers' perception of test smells and proposed tools 18–20 and thresholds to analyze them 10,21,22 . Most of those studies use customized datasets, often not shared publicly, making it hard to compare the outcomes as the data come from different projects, versions, and contexts.…”

“…Therefore, we introduce the TSSM (Test Smells and Structural Metrics) dataset † TSSM encompasses 19 test smells detected with the JNose Test ‡ , 23 tool and 45 structural metrics of test code and production code calculated with the CK metrics § , 24 tool from 13,703 Java open‐source projects hosted on GitHub ¶ . We used the TSSM dataset through a replication study to investigate (i) the diffusion of test smells in JUnit test suites, (ii) the co‐occurrences among test smells, and (iii) the association between test smells and structural metrics of test code.…”