On the Lack of Consensus Among Technical Debt Detection Tools

Abstract

A vigorous and growing set of technical debt analysis tools have been developed in recent years-both research tools and industrial products-such as Structure 101, SonarQube, and DV8. Each of these tools identifies problematic files using their own definitions and measures. But to what extent do these tools agree with each other in terms of the files that they identify as problematic? If the top-ranked files reported by these tools are largely consistent, then we can be confident in using any of these tools. Otherwise, a problem of accuracy arises. In this paper, we report the results of an empirical study analyzing 10 projects using multiple tools. Our results show that: 1) these tools report very different results even for the most common measures, such as size, complexity, file cycles, and package cycles. 2) These tools also differ dramatically in terms of the set of problematic files they identify, since each implements its own definitions of "problematic". After normalizing by size, the most problematic file sets that the tools identify barely overlap. 3) Our results show that code-based measures, other than size and complexity, do not even moderately correlate with a file's change-proneness or errorproneness. In contrast, co-change-related measures performed better. Our results suggest that, to identify files with true technical debt-those that experience excessive changes or bugsco-change information must be considered. Code-based measures are largely ineffective at pinpointing true debt. Finally, this study reveals the need for the community to create benchmarks and data sets to assess the accuracy of software analysis tools in terms of commonly used measures.

On the evolution and impact of Architectural Smells -- An industrial case study

“…Another threat concerns the detection of the smells considered in this project, which depend on the implementation offered by Arcan. Lefever et al [Lefever et al, 2021] have shown that different tools for technical debt measurement (including DV8, CAST, and SonarQube, but not Arcan) have divergent, if not conflicting, results regarding which files are problematic in a system. This is due to the fact that different tools make different assumptions, use different definitions of a smell, and have different implementations of how to detect a smell [Lefever et al, 2021].…”

“…Lefever et al [Lefever et al, 2021] have shown that different tools for technical debt measurement (including DV8, CAST, and SonarQube, but not Arcan) have divergent, if not conflicting, results regarding which files are problematic in a system. This is due to the fact that different tools make different assumptions, use different definitions of a smell, and have different implementations of how to detect a smell [Lefever et al, 2021]. Therefore, we can only state that our quantitative results obtained through Arcan may not be fully comparable with the results obtained by other tools.…”

“…Therefore, we can only state that our quantitative results obtained through Arcan may not be fully comparable with the results obtained by other tools. However, this would be the case even if we used any other tool, as shown by Lefever et al [Lefever et al, 2021]. Having said that, this threat can be considered partially mitigated, as the definitions of each architectural smell used by Arcan are based on independent, previous work.…”

On the evolution and impact of architectural smells—an industrial case study

“…Another threat concerns the detection of the smells considered in this project, which depend on the implementation offered by ARCAN. Lefever et al (2021) have shown that different tools for technical debt measurement (including DV8, CAST, and SonarQube, but not ARCAN) have divergent, if not conflicting, results regarding which files are problematic in a system. This is due to the fact that different tools make different assumptions, use different definitions of a smell, and have different implementations of how to detect a smell (Lefever et al 2021).…”

“…Lefever et al (2021) have shown that different tools for technical debt measurement (including DV8, CAST, and SonarQube, but not ARCAN) have divergent, if not conflicting, results regarding which files are problematic in a system. This is due to the fact that different tools make different assumptions, use different definitions of a smell, and have different implementations of how to detect a smell (Lefever et al 2021). Therefore, we can only state that our quantitative results obtained through ARCAN may not be fully comparable with the results obtained by other tools.…”