On the performance of method-level bug prediction: A negative result

Abstract

Bug prediction is aimed at identifying software artifacts that are more likely to be defective in the future. Most approaches defined so far target the prediction of bugs at class/file level. Nevertheless, past research has provided evidence that this granularity is too coarse-grained for its use in practice. As a consequence, researchers have started proposing defect prediction models targeting a finer granularity (particularly method-level granularity), providing promising evidence that it is possible to operate at this level. Particularly, models mixing product and process metrics provided the best results. We present a study in which we first replicate previous research on method-level bug-prediction, by using different systems and timespans. Afterwards, based on the limitations of existing research, we (1) re-evaluate method-level bug prediction models more realistically and (2) analyze whether alternative features based on textual aspects, code smells, and developer-related factors can be exploited to improve method-level bug prediction abilities. Key results of our study include that (1) the performance of the previously proposed models, tested using the same strategy but on different systems/timespans, is confirmed; but, (2) when evaluated with a more practical strategy, all the models show a dramatic drop in performance, with results close to that of a random classifier. Finally, we find that (3) the contribution of alternative features within such models is limited and unable to improve the prediction capabilities significantly. As a consequence, our replication and negative results indicate that method-level bug prediction is still an open challenge.

An Empirical Study on Maintainable Method Size in Java

“…Internal metrics, also known as code metrics, are easy to collect and are available throughout the SDLC. Therefore, it has been a holy grail to the research community to understand software maintenance using different code metrics [18,46,52,53,62,86]. Consequently, a significant numbers of code metrics have been proposed and studied over the last forty years [48].…”

“…McCabe [52], or cyclomatic complexity, is a measure of the number of independent paths in a source code component. The assumption is that a source code component with high McCabe score would be hard to maintain; it would be more change-and bugprone [5,6,21,47,61,62,68,84,93]. Another popular code metric is the Halstead metric, which is mainly based on the number of operators and operands in a code component [7,21,35,43,63].…”

“…Two of the most widely adopted maintenance indicators are change-proneness [26,69] and bug-proneness [38,54,62,85,94]. These metrics suggest that code components (e.g., methods, classes, or modules) that are bug-or change-prone are generally expensive to maintain.…”

“…To address this shortcoming, researchers have investigated static code metrics to learn how these metrics alias with change-and bugproneness (e.g., [26,43,62,93]). These metrics include McCabe [52],…”

“…Unfortunately, there is no evidence-based guidelines that developers can follow to optimize the size of their components. In this paper we focus on method level granularity to measure size (e.g., in contrast to class-, or module-level granularities), because method level granularity is desired by both developers and researchers [29,62]. There is not currently broad consensus for how long a method should be; for example, consider 'What is the ideal length of a method for you?'…”