Fuzzing class specifications

Molina, Facundo; d’Amorim, Marcelo; Aguirre, Nazareno

doi:10.1145/3510003.3510120

Cited by 16 publications

(25 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To answer this question we use a dataset composed by manually written assertions (ground-truth) that was recently used for evaluating SpecFuzzer tool [26], a state-of-the-art specification inference technique. Particularly, we select 4 manually written assertions that were mistakenly discarded by SpecFuzzer, since they do not kill any mutant.…”

Section: Research Questionsmentioning

confidence: 99%

“…For the assertion assessment analysis, we use the dataset from SpecFuzzer, a specification inference technique recently introduced by Molina et al [26], that includes (41) assertions manually written by developers. Each subject contains the source code, the test suite used during the inference process, and the set of manually written expected assertions.…”

Section: Experimental Setup a Faults And Assertions (Ground-truth)mentioning

confidence: 99%

“…Particularly, we focus on 4 methods of the dataset (StackAr.pop, StackAr.topAndPop, Angle.getTurn and Composite.addChild) in which 6 assertions from the ground-truth are discarded since they do not kill any mutant (cf. [26,Table 4]). We study whether µBERT can help SpecFuzzer in selecting the discarded assertions, and compare with PiTest.…”

Section: Experimental Setup a Faults And Assertions (Ground-truth)mentioning

confidence: 99%

“…To answer RQ2, we start by generating mutants with µBERT and PiTest for the four methods under analysis. Then we run the inference tool, SpecFuzzer [26], to obtain the a set of valid assertions for the method of interest (i.e., never falsified by the test suite). SpecFuzzer then performs a mutation analysis on the inferred assertions, and discards the ones that do not kill any mutant.…”

Section: B Experimental Proceduresmentioning

confidence: 99%

“…Assertion inference: We study the usefulness of µBERT's mutants in the context of program assertion inference techniques, that use mutants to rank and discard candidate assertions [16] (typically, assertions that kill more mutants are preferred among others, and assertions not killing any mutant are discarded). In particular, we focus on the 4 cases recently reported in [26] in which traditional mutation testing did not perform well. We show that µBERT can complement and contribute with interesting mutants than can help in improving…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

µBert: Mutation Testing using Pre-Trained Language Models

Degiovanni

Papadakis

2022

2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

View full text Add to dashboard Cite

We introduce µBERT, a mutation testing tool that uses a pre-trained language model (CodeBERT) to generate mutants. This is done by masking a token from the expression given as input and using CodeBERT to predict it. Thus, the mutants are generated by replacing the masked tokens with the predicted ones. We evaluate µBERT on 40 real faults from Defects4J and show that it can detect 27 out of the 40 faults, while the baseline (PiTest) detects 26 of them. We also show that µBERT can be 2 times more cost-effective than PiTest, when the same number of mutants are analysed. Additionally, we evaluate the impact of µBERT's mutants when used by program assertion inference techniques, and show that they can help in producing better specifications. Finally, we discuss about the quality and naturalness of some interesting mutants produced by µBERT during our experimental evaluation.

show abstract

Section: Research Questionsmentioning

confidence: 99%

Section: Experimental Setup a Faults And Assertions (Ground-truth)mentioning

confidence: 99%

Section: Experimental Setup a Faults And Assertions (Ground-truth)mentioning

confidence: 99%

Section: B Experimental Proceduresmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

µBert: Mutation Testing using Pre-Trained Language Models

Degiovanni

Papadakis

2022

2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

View full text Add to dashboard Cite

show abstract

Comprehending Object State via Dynamic Class Invariant Learning

Boockmann,

Lüttgen

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Maintaining software is cumbersome when method argument constraints are undocumented. To reveal them, previous work learned preconditions from exemplary valid and invalid method arguments. In practice, it would be highly beneficial to know class invariants, too, because functionality added during software maintenance must not break them. Even more so than method preconditions, class invariants are rarely documented and often cannot completely be inferred automatically, especially for objects exhibiting complex state such as dynamic data structures.This paper presents a novel dynamic approach to learning class invariants, thereby complementing related work on learning method preconditions. We automatically synthesize assertions from an adjustable assertion grammar to distinguish valid and invalid objects. While random walks generate valid objects, a combination of bounded-exhaustive testing techniques and behavioral oracles yield invalid objects. The utility of our approach for code comprehension and software maintenance is demonstrated by comparing our learned invariants to documented invariant validation methods found in real-world Java classes and to the invariants detected by the Daikon tool.

show abstract

Enchanting Program Specification Synthesis by Large Language Models Using Static Analysis and Program Verification

Wen,

Cao,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Formal verification provides a rigorous and systematic approach to ensure the correctness and reliability of software systems. Yet, constructing specifications for the full proof relies on domain expertise and non-trivial manpower. In view of such needs, an automated approach for specification synthesis is desired. While existing automated approaches are limited in their versatility, i.e., they either focus only on synthesizing loop invariants for numerical programs, or are tailored for specific types of programs or invariants. Programs involving multiple complicated data types (e.g., arrays, pointers) and code structures (e.g., nested loops, function calls) are often beyond their capabilities. To help bridge this gap, we present AutoSpec, an automated approach to synthesize specifications for automated program verification. It overcomes the shortcomings of existing work in specification versatility, synthesizing satisfiable and adequate specifications for full proof. It is driven by static analysis and program verification, and is empowered by large language models (LLMs). AutoSpec addresses the practical challenges in three ways: (1) driving AutoSpec by static analysis and program verification, LLMs serve as generators to generate candidate specifications, (2) programs are decomposed to direct the attention of LLMs, and (3) candidate specifications are validated in each round to avoid error accumulation during the interaction with LLMs. In this way, AutoSpec can incrementally and iteratively generate satisfiable and adequate specifications. The evaluation shows its effectiveness and usefulness, as it outperforms existing works by successfully verifying 79% of programs through automatic specification synthesis, a significant improvement of 1.592x. It can also be successfully applied to verify the programs in a real-world X509-parser project.

show abstract

Fuzzing class specifications

Cited by 16 publications

References 39 publications

µBert: Mutation Testing using Pre-Trained Language Models

µBert: Mutation Testing using Pre-Trained Language Models

Comprehending Object State via Dynamic Class Invariant Learning

Enchanting Program Specification Synthesis by Large Language Models Using Static Analysis and Program Verification

Contact Info

Product

Resources

About