Problems with SZZ and features: An empirical study of the state of practice of defect prediction data collection

Abstract

Context The SZZ algorithm is the de facto standard for labeling bug fixing commits and finding inducing changes for defect prediction data. Recent research uncovered potential problems in different parts of the SZZ algorithm. Most defect prediction data sets provide only static code metrics as features, while research indicates that other features are also important. Objective We provide an empirical analysis of the defect labels created with the SZZ algorithm and the impact of commonly used features on results. Method We used a combination of manual validation and adopted or improved heuristics for the collection of defect data. We conducted an empirical study on 398 releases of 38 Apache projects. Results We found that only half of the bug fixing commits determined by SZZ are actually bug fixing. If a six-month time frame is used in combination with SZZ to determine which bugs affect a release, one file is incorrectly labeled as defective for every file that is correctly labeled as defective. In addition, two defective files are missed. We also explored the impact of the relatively small set of features that are available in most defect prediction data sets, as there are multiple publications that indicate that, e.g., churn related features are important for defect prediction. We found that the difference of using more features is not significant. Conclusion Problems with inaccurate defect labels are a severe threat to the validity of the state of the art of defect prediction. Small feature sets seem to be a less severe threat.

On the validity of pre-trained transformers for natural language processing in the software engineering domain

“…Thus, we can not only compare pre-trained models to each other, but also their benefit over simpler models that can be trained in minutes without any special hardware. Second, the task is hard and often not solved correctly by humans, unless careful manual validation is used [43], [44], [45].…”

“…We suggest to use the five projects from Herzig et al [44] and the 38 projects from Herbold et al [45] together and conduct a leave-one-project-out cross validation experiment. This means that one project is used for testing and the remaining 42 projects for the training of the classifier, i.e., of the a fastText model as baseline and for the fine-tuning of the BERT models.…”

Are automated static analysis tools worth it? An investigation into relative warning density and external software quality

“…Our study subjects consist of 23 Java projects under the umbrella of the Apache Software Foundation 3 previously collected by (Herbold et al, 2020). Table 1 contains the list of our study subjects.…”

“…The ITS usually has a kind of label or type to distinguish bugs from other issues, e.g., feature requests. However, research shows that this label is often incorrect, e.g., Antoniol et al (2008), Herzig et al (2013b) and Herbold et al (2020). Moreover, with this kind of software evolution research, we are interested in bugs existing in the software and not bugs which occur because of external factors, e.g., new environments or dependency upgrades.…”

“…This was achieved in a prior publication (Herbold et al, 2020) which served as the base for the publication which data we use in this article (Herbold et al, 2021). In (Herbold et al, 2021) a detailed untangling is performed by four different persons for each change that fixes a bug that meets our criteria.…”