An empirical study of issue-link algorithms: which issue-link algorithms should we use?

Abstract

The accuracy of the SZZ algorithm is pivotal for just-in-time defect prediction because most prior studies have used the SZZ algorithm to detect defect-inducing commits to construct and evaluate their defect prediction models. The SZZ algorithm has two phases to detect defect-inducing commits: (1) linking issue reports in an issue-tracking system to possible defect-fixing commits in a version control system by using an issue-link algorithm (ILA); and (2) tracing the modifications of defect-fixing commits back to possible defect-inducing commits. Researchers and practitioners can address the second phase by using existing solutions such as a tool called . In contrast, although various ILAs have been proposed for the first phase, no large-scale studies exist in which such ILAs are evaluated under the same experimental conditions. Hence, we still have no conclusions regarding the best-performing ILA for the first phase. In this paper, we compare 10 ILAs collected from our systematic literature study with regards to the accuracy of detecting defect-fixing commits. In addition, we compare the defect prediction performance of ILAs and their combinations that can detect defect-fixing commits accurately. We conducted experiments on five open-source software projects. We found that all ILAs and their combinations prevented the defect prediction model from being affected by missing defect-fixing commits. In particular, the combination of a natural language text similarity approach, Phantom heuristics, a random forest approach, and a support vector machine approach is the best way to statistically significantly reduced the absolute differences from the ground-truth defect prediction performance. We summarized the guidelines to use ILAs as our recommendations.

A Systematic Literature Review of Issue-Based Requirement Traceability

“…Studies sought to improve the low accuracy of the existing techniques for generating, recovering, or maintaining trace links [25], [36], [37], [46], [49], [50], [54], [56]. Researchers attributed the low accuracy to error-prone links, semantic gaps, and large source files.…”

“…A total of 18 studies linked issue reports to commits. [16], [25], [27], [29], [34]- [38], [42], [44], [48], [51], [54]- [56], [58], [62]. Among them, eleven studies focused on recovering the missing trace links between issue reports and commits [16], [27], [29], [36]- [38], [44], [51], [56], [58], [62].…”

“…[16], [25], [27], [29], [34]- [38], [42], [44], [48], [51], [54]- [56], [58], [62]. Among them, eleven studies focused on recovering the missing trace links between issue reports and commits [16], [27], [29], [36]- [38], [44], [51], [56], [58], [62]. Two studies focused on generating trace links between issue reports written in natural language and commits written in programming language [25], [42].…”

“…Five studies selected various open-source projects. [27], [49], [53], [56], [62]. The study in [53] used 803 issue reports from 15 open-source projects.…”

“…The study in [27] selected 6 projects, where the study collected trace data from project management systems, issue tracking systems, and code management systems. The study in [56] selected 5 projects to comparatively evaluate 10 issue-linking algorithms. The study in [49] selected 4 projects and extracted 100 consecutive issues per project.…”