On the efficiency of test suite based program repair

Liu, Kui; Wang, Shangwen; Koyuncu, Anil; Kim, Kisub; Bissyandé, Tegawendé F.; Kim, Dong–Sun; Wu, Peng; Klein, Jacques; Mao, Xiaoguang; Traon, Yves Le

doi:10.1145/3377811.3380338

Cited by 92 publications

(65 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, the tools validate the generated patches by calculating the number of tests that pass when a suggested patch is applied to the program under examination. We note that in this category we also include tools that use template patterns since according to Liu et al [10] technically these tools work similarly to "Heuristic-based". Additionally, "Constraint-based" tools proceed by constructing a repair constraint that the patched code should satisfy.…”

Section: Automatic Program Repairmentioning

confidence: 99%

“…Specifically, our study compares the ability of 14 Java test-suite-based repair tools to automatically detect and fix API misuses in client Java programs. According to Liu et al [10], we categorise these tools into less recent (introduced before 2018) and more recent (introduced after 2018) tools. In our study, we compare 11 earlier tools (first 11 tools in Table 4) and three more recent tools (last three tools in Table 4).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating Automatic Program Repair Capabilities to Repair API Misuses

Kechagia

Mechtaev

Sarro

et al. 2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

API misuses are well-known causes of software crashes and security vulnerabilities. However, their detection and repair is challenging given that the correct usages of (third-party) APIs might be obscure to the developers of client programs. This paper presents the first empirical study to assess the ability of existing automated bug repair tools to repair API misuses, which is a class of bugs previously unexplored. Our study examines and compares 14 Java test-suite-based repair tools (11 proposed before 2018, and three afterwards) on a manually curated benchmark (APIREPBENCH) consisting of 101 API misuses. We develop an extensible execution framework (APIARTY) to automatically execute multiple repair tools. Our results show that the repair tools are able to generate patches for 28% of the API misuses considered. While the 11 less recent tools are generally fast (the median execution time of the repair attempts is 3.87 minutes and the mean execution time is 30.79 minutes), the three most recent are less efficient (i.e., 98% slower) than their predecessors. The tools generate patches for API misuses that mostly belong to the categories of missing null check, missing value, missing exception, and missing call. Most of the patches generated by all tools are plausible (65%), but only few of these patches are semantically correct to human patches (25%). Our findings suggest that the design of future repair tools should support the localisation of complex bugs, including different categories of API misuses, handling of timeout issues, and ability to configure large software projects. Both APIREPBENCH and APIARTY have been made publicly available for other researchers to evaluate the capabilities of repair tools on detecting and fixing API misuses.

show abstract

Section: Automatic Program Repairmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Evaluating Automatic Program Repair Capabilities to Repair API Misuses

Kechagia

Mechtaev

Sarro

et al. 2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…We investigate the first research question by defining two metrics that (1) evaluate the actual performance that a given APR's patch generation component can achieve when the exact bug-fixing positions are given; and (2) estimate the impact on repair performance of the false positives induced by the APR's implementation of a given fault localization technique. Note that, all data for this research question are from our previous empirical study (Liu et al, 2020) on the efficiency of APR tools.…”

Section: Rq1: Apr Pipeline Implementation Biasesmentioning

confidence: 99%

“…A correct patch is a valid patch that actually fixes the bug (Qi et al, 2015) (i.e., is not simply overfitted to the test suite). Correctness is generally determined manually (Xiong et al, 2017;Liu et al, 2020Liu et al, , 2019bJiang et al, 2018;Wen et al, 2018).…”

Section: Rq2: Apr Efficiency Assessment Biasesmentioning

confidence: 99%

See 1 more Smart Citation

A critical review on the evaluation of automated program repair systems

Kui

Koyuncu

et al. 2021

Journal of Systems and Software

Self Cite

View full text Add to dashboard Cite

Automated Program Repair (APR) has attracted significant attention from software engineering research and practice communities in the last decade. Several teams have recorded promising performance in fixing real bugs and there is a race in the literature to fix as many bugs as possible from established benchmarks. Gradually, repair performance of APR tools in the literature has gone from being evaluated with a metric on the number of generated plausible patches to the number of correct patches. This evolution is necessary after a study highlighting the overfitting issue in test suite-based automatic patch generation. Simultaneously, some researchers are also insisting on providing time cost in the repair scenario as a metric for comparing state-of-the-art systems. In this paper, we discuss how the latest evaluation metrics of APR systems could be biased. Since design decisions (both in approach and evaluation setup) are not always fully disclosed, the impact on repair performance is unknown and computed metrics are often misleading. To reduce notable biases of design decisions in program repair approaches, we conduct a critical review on the evaluation of patch generation systems and propose eight evaluation metrics for fairly assessing the performance of APR tools. Eventually, we show with experimental data on 11 baseline program repair systems that the proposed metrics allow to highlight some caveats in the literature. We expect wide adoption of these metrics in the community to contribute to boosting the development of practical, and reliably performable program repair tools.

show abstract

Enhanced evolutionary automated program repair by finer‐granularity ingredients and better search algorithms

Wang,

Liu,

Lin

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

SummaryBug repair is time consuming and tedious, which hampers software maintenance. To alleviate the burden, automated program repair (APR) is proposed and has been fruitful in the last decade. Evolutionary repair is the seminal work of this field and proliferated a family of approaches. The performance of evolutionary repair approaches is affected by two main factors: (1) search space, which defines all possible patches, and (2) search algorithms, which navigate the space. Although recent approaches have achieved remarkable progress, the main challenges of the two factors still remain. On one hand, the different kinds of search space are very coarse for containing correct patches. On the other hand, the search process guided by genetic algorithms is inefficient in finding the correct patches in an appropriate time budget. In this paper, we propose MicroRepair, a new evolutionary repair approach to address the two challenges. Rather than finding statement‐level patches like existing genetic repair approaches, MicroRepair enlarges the search space by breaking the statements into finer‐granularity ingredients that consist of AST leaves. As the search space grows exponentially, the former search algorithms may become inefficient in navigating the larger space. We utilize the best multiobjective search algorithm selected from our empirical comparison of a set of search algorithms. At last, we find redundancies search in the existing genetic process, and we further design a history‐aware search strategy to accelerate the process.We evaluated MicroRepair on 224 bugs of real‐world from the benchmark Defects4J and compared it with several state‐of‐the‐art repair approaches. The evaluation results show that MicroRepair correctly repaired 26 bugs with a precision of 62%, which significantly outperforms the state‐of‐the‐art evolutionary APR approaches in terms of precision. Moreover, the history‐aware search boosts the repair execution speed by 4% on average.

show abstract

On the efficiency of test suite based program repair

Abstract: Test-based automated program repair has been a prolific field of research in software engineering in the last decade. Many approaches have indeed been proposed, which leverage test suites as a weak, but affordable, approximation to program specifications. Although the

Cited by 92 publications

References 69 publications

Evaluating Automatic Program Repair Capabilities to Repair API Misuses

Evaluating Automatic Program Repair Capabilities to Repair API Misuses

A critical review on the evaluation of automated program repair systems

Enhanced evolutionary automated program repair by finer‐granularity ingredients and better search algorithms

Contact Info

Product

Resources

About