CoCoNuT: combining context-aware neural translation models using ensemble for program repair

Lutellier, Thibaud; Pham, Hūng Viêt; Pang, Lawrence; Li, Yitong; Wei, Moshi; Tan, Lin

doi:10.1145/3395363.3397369

Cited by 243 publications

(285 citation statements)

References 64 publications

Supporting

Mentioning

255

Contrasting

Order By: Relevance

“…where A is a set of APR tools and UbRP(apr ∈ A) gives a nonnegative integer value (∈ Z * ) (i.e., # of correctly fixed bugs). Such metric is first proposed as the repair performance of APR tools with perfect fault localization in our previous study (Liu et al, 2019a) and has been adopted in the community (Liu et al, 2019b,c;Lutellier et al, 2020;Chen et al, 2020;Chen et al, 2019).…”

Section: Upper Bound Repair Performancementioning

confidence: 99%

A critical review on the evaluation of automated program repair systems

Kui

Koyuncu

et al. 2021

Journal of Systems and Software

View full text Add to dashboard Cite

Automated Program Repair (APR) has attracted significant attention from software engineering research and practice communities in the last decade. Several teams have recorded promising performance in fixing real bugs and there is a race in the literature to fix as many bugs as possible from established benchmarks. Gradually, repair performance of APR tools in the literature has gone from being evaluated with a metric on the number of generated plausible patches to the number of correct patches. This evolution is necessary after a study highlighting the overfitting issue in test suite-based automatic patch generation. Simultaneously, some researchers are also insisting on providing time cost in the repair scenario as a metric for comparing state-of-the-art systems. In this paper, we discuss how the latest evaluation metrics of APR systems could be biased. Since design decisions (both in approach and evaluation setup) are not always fully disclosed, the impact on repair performance is unknown and computed metrics are often misleading. To reduce notable biases of design decisions in program repair approaches, we conduct a critical review on the evaluation of patch generation systems and propose eight evaluation metrics for fairly assessing the performance of APR tools. Eventually, we show with experimental data on 11 baseline program repair systems that the proposed metrics allow to highlight some caveats in the literature. We expect wide adoption of these metrics in the community to contribute to boosting the development of practical, and reliably performable program repair tools.

show abstract

Section: Upper Bound Repair Performancementioning

confidence: 99%

A critical review on the evaluation of automated program repair systems

Kui

Koyuncu

et al. 2021

Journal of Systems and Software

View full text Add to dashboard Cite

show abstract

“…The existence of defects will affect program quality seriously, but the existing program automatic repair methods and tools [16][17][18][19][20][21][22][23][24][25][26][27] cannot guarantee that the repaired program is correct. For example, the automatic repair tool GenProg [5] repairs the defect in Figure 2 mistakenly.…”

Section: Automatic Program Repair Without Side Effectsmentioning

confidence: 99%

Automated Program-Semantic Defect Repair and False-Positive Elimination without Side Effects

Dong¹,

Wu²,

Pang³

et al. 2020

Symmetry

View full text Add to dashboard Cite

The alarms of the program-semantic defect-detection report based on static analysis include defects and false positives. The repair of defects and the elimination of false positives are time-consuming and laborious, and new defects may be introduced in the process. To solve these problems, the safe constraints interval of related variables and methods are proposed for the semantic defects in the program, and proposes a functionally equivalent no-side-effect program-semantic defect repair and false-positive elimination strategy based on the test-equivalence theory. This paper realizes the automatic repair of the typical semantic defects of Java programs and the automatic elimination of false positives by adding safe constraint patches. After the repair, the program functions are equivalent and the status of each program point is within the safety range, so that the functions before and after the defect repair are consistent, and the functions and semantics before and after the false positives are eliminated. We have evaluated our approach by repairing 5 projects; our results show that the repair strategy does not require manual confirmation of alarms, automated repair of the program effectively, shortened the repair time greatly, and ensured the correctness of the program after the repair.

show abstract

“…Such ineffectiveness is evident as demonstrated by data. For example, up to 67%-97% of patches generated by the state-ofthe-art NMT-based APR models [19], [20] are uncompilable, wasting valuable resources on incorrect patches. Figure 1 shows a bug in QuixBugs and some of the top-ranked patches generated by CoCoNuT [19].…”

Section: Introductionmentioning

confidence: 99%

“…For example, up to 67%-97% of patches generated by the state-ofthe-art NMT-based APR models [19], [20] are uncompilable, wasting valuable resources on incorrect patches. Figure 1 shows a bug in QuixBugs and some of the top-ranked patches generated by CoCoNuT [19]. All of these patches are uncompilable, because they call methods with wrong parameters, invoke undeclared variables, or contain mismatched parenthe-sis.…”

Section: Introductionmentioning

confidence: 99%

“…Yet with a limited vocabulary size, their search spaces do not contain all correct fixes. SequenceR [20] uses a small vocabulary and shirks this complexity to a later reconstruction stage, while CoCoNuT [19] uses a vocabulary of more than 130,000 tokens but still suffers from the outof-vocabulary (OOV, i.e., an NMT model cannot recognize or generate a token) problem, resulting in its search space that still misses correct fixes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CURE: Code-Aware Neural Machine Translation for Automatic Program Repair

Jiang

Lutellier

Tan

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

Self Cite

189

181

View full text Add to dashboard Cite

Automatic program repair (APR) is crucial to improve software reliability. Recently, neural machine translation (NMT) techniques have been used to fix software bugs automatically. While promising, these approaches have two major limitations. Their search space often does not contain the correct fix, and their search strategy ignores software knowledge such as strict code syntax. Due to these limitations, existing NMT-based techniques underperform the best template-based approaches. We propose CURE, a new NMT-based APR technique with three major novelties. First, CURE pre-trains a programming language (PL) model on a large software codebase to learn developer-like source code before the APR task. Second, CURE designs a new code-aware search strategy that finds more correct fixes by focusing on compilable patches and patches that are close in length to the buggy code. Finally, CURE uses a subword tokenization technique to generate a smaller search space that contains more correct fixes.Our evaluation on two widely-used benchmarks shows that CURE correctly fixes 57 Defects4J bugs and 26 QuixBugs bugs, outperforming all existing APR techniques on both benchmarks.

show abstract

CoCoNuT: combining context-aware neural translation models using ensemble for program repair

Cited by 243 publications

References 64 publications

A critical review on the evaluation of automated program repair systems

A critical review on the evaluation of automated program repair systems

Automated Program-Semantic Defect Repair and False-Positive Elimination without Side Effects

CURE: Code-Aware Neural Machine Translation for Automatic Program Repair

Contact Info

Product

Resources

About