A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark

Abstract

Automatic program repair papers tend to repeatedly use the same benchmarks. This poses a threat to the external validity of the findings of the program repair research community. In this paper, we perform an automatic repair experiment on a benchmark called QuixBugs that has never been studied in the context of program repair. In this study, we report on the characteristics of QuixBugs, and study five repair systems, Arja, Astor, Nopol, NPEfix and RSRepair, which are representatives of generate-and-validate repair techniques and synthesis repair techniques. We propose three patch correctness assessment techniques to comprehensively study overfitting and incorrect patches. Our key results are: 1) 15 / 40 buggy programs in the QuixBugs can be repaired with a test-suite adequate patch; 2) a total of 64 plausible patches for those 15 buggy programs in the QuixBugs are present in the search space of the considered tools; 3) the three patch assessment techniques discard in total 33 / 64 patches that are overfitting. This sets a baseline for future research of automatic repair on QuixBugs. Our experiment also highlights the major properties and challenges of how to perform automated correctness assessment of program repair patches. All experimental results are publicly available on Github in order to facilitate future research on automatic program repair.

Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts

“…For instance, in the paper that jGenProg [27] is presented, there is an evaluation on Defects4J: this evaluation has no citation in the second column of the table because the evaluation is in jGenProg's paper. Later, it was evaluated again on Defects4J [26] and also on QuixBugs [48], which contain citations of the empirical evaluation papers in the table. The table also presents additional information on the evaluations, which are the number of bugs given as input to the repair tools, and the number of bugs for which the tools generated a test-suite adequate patch (i.e.…”

“…They also found that a small number of bugs (9/47) could be repaired with a test-suite adequate patch that is also correct. Ye et al [48] presented a study where nine repair tools were executed on the bugs from QuixBugs. They used automatically generated test cases based on the human-written patches to identify incorrect patches generated by the repair tools.…”

“…[50]), or when a dedicated full contribution on evaluating existing repair tools is reported (e.g. [26,33,48]). The evaluations consist of four main aspects in general: 1) [benchmark] the selection of benchmarks of bugs; 2) [execution] the collection of data by executing repair tools on the selected bugs; 3) [observed aspect] an investigation on the effectiveness of the repair approach regarding some criteria (e.g.…”

How Different Is It Between Machine-Generated and Developer-Provided Patches? : An Empirical Study on the Correct Patches Generated by Automated Program Repair Techniques

“…The other aspect is test adequacy of the buggy class. We use line coverage and branch coverage to measure it as existing studies do [49,50]. Our intuition is that the test quality measured by line and branch coverages is related with the type of correct patches generated for this bug since existing studies have shown that the correctness (i.e., plausible, overfitting, or correct) of APR generated patches has strong correlation with the test quality [32,34,36].…”

“…The data of patch complexity is from the previous study [48] in which the characteristics of each bug in Defects4J have been analyzed. The data of test adequacy is calculated by Cobertura 1 which is a free Java tool being widely-used in recent studies [49,50]. If different types of patches are generated for the same bug, the data of this bug is added into all the relevant types for analysis.…”