On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools

Papotti, Aurora; Paramitha, Ranindya; Massacci, Fabio

doi:10.48550/arxiv.2209.07211

Cited by 2 publications

(1 citation statement)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human-competitiveness As mentioned by Monperrus et al (2019), there are two criteria for the tools to be human-competitive: (1) the tool generates the patch faster than the human developer, and (2) the human developer accepts the generated patch. The second criterion is out of scope for this paper (it has been studied by our prior work Papotti et al 2022). For the first criterion, most of the tools in our study (TBar, Cardumen, Kali-A, jKali, jMutRepair) halt immediately after the first E2E tested patch is found.…”

Section: The Technical Implementations Of Arja and Genprog-a Cause Ov...mentioning

confidence: 99%

APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities

Bui,

Paramitha,

et al. 2023

Empir Software Eng

View full text Add to dashboard Cite

Security vulnerability fixes could be a promising research avenue for Automated Program Repair (APR) techniques. In recent years, APR tools have been thoroughly developed for fixing generic bugs. However, the area is still relatively unexplored when it comes to fixing security bugs or vulnerabilities. In this paper, we evaluate nine state-of-the-art APR tools and one vulnerability-specific repair tool. In particular, we investigate their ability to generate patches for 79 real-world Java vulnerabilities in the Vul4J dataset, as well as the level of trustworthiness of these patches. We evaluate the tools with respect to their ability to generate security patches that are (i) testable, (ii) having the positive effect of closing the vulnerability, and (iii) not having side effects from a functional point of view. Our results show that the evaluated APR tools were able to generate testable patches for around 20% of the considered vulnerabilities. On average, nearly 73% of the testable patches indeed eliminate the vulnerabilities, but only 44% of them could actually fix security bugs while maintaining the functionalities. To understand the root cause of this phenomenon, we conduct a detailed comparative study of the general bug fix patterns in Defect4J and the vulnerability fix patterns in ExtraVul (which we extend from Vul4J). Our investigation shows that, although security patches are short in terms of lines of code, they contain unique characteristics in their fix patterns compared to general bugs. For example, many security fixes require adding method calls. These method calls contain specific input validation-related keywords, such as encode, normalize, and trim. In this regard, our study suggests that additional repair patterns should be implemented for existing APR tools to fix more types of security vulnerabilities.

show abstract

Section: The Technical Implementations Of Arja and Genprog-a Cause Ov...mentioning

confidence: 99%

APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities

Bui,

Paramitha,

et al. 2023

Empir Software Eng

View full text Add to dashboard Cite

show abstract

On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools

Papotti,

Paramitha,

Massacci

2024

Empir Software Eng

View full text Add to dashboard Cite

Objective We investigated whether (possibly wrong) security patches suggested by Automated Program Repairs (APR) for real world projects are recognized by human reviewers. We also investigated whether knowing that a patch was produced by an allegedly specialized tool does change the decision of human reviewers. Method We perform an experiment with $$n= 72$$ n = 72 Master students in Computer Science. In the first phase, using a balanced design, we propose to human reviewers a combination of patches proposed by APR tools for different vulnerabilities and ask reviewers to adopt or reject the proposed patches. In the second phase, we tell participants that some of the proposed patches were generated by security-specialized tools (even if the tool was actually a ‘normal’ APR tool) and measure whether the human reviewers would change their decision to adopt or reject a patch. Results It is easier to identify wrong patches than correct patches, and correct patches are not confused with partially correct patches. Also patches from APR Security tools are adopted more often than patches suggested by generic APR tools but there is not enough evidence to verify if ‘bogus’ security claims are distinguishable from ‘true security’ claims. Finally, the number of switches to the patches suggested by security tool is significantly higher after the security information is revealed irrespective of correctness. Limitations The experiment was conducted in an academic setting, and focused on a limited sample of popular APR tools and popular vulnerability types.

show abstract

On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools

Cited by 2 publications

References 37 publications

APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities

APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities

On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools

Contact Info

Product

Resources

About