Semantic bug seeding: a learning-based approach for creating realistic bugs

Patra, Jibesh; Pradel, Michael

doi:10.1145/3468264.3468623

Cited by 47 publications

(14 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use six classes of denaturalizing transformations. These transformations are motivated by prior work on functional reasoning about source code [22,26,27] and semantic bug-seeding [47]. Figure 2 show the details.…”

Section: Designing Transformation Rulesmentioning

confidence: 99%

NatGen: Generative pre-training by "Naturalizing" source code

Chakraborty¹,

Ahmed²,

Ding³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-trained Generative Language models (e.g., PLBART, CodeT5, SPT-Code) for source code yielded strong results on several tasks in the past few years, including code generation and translation. These models have adopted varying pre-training objectives to learn statistics of code construction from very large-scale corpora in a self-supervised fashion; the success of pre-trained models largely hinges on these pre-training objectives. This paper proposes a new pre-training objective, "Naturalizing" of source code, exploiting code's bimodal, dual-channel (formal & natural channels) nature. Unlike natural language, code's bimodal, dual-channel nature allows us to generate semantically equivalent code at scale. We introduce six classes of semantic preserving transformations to introduce un-natural forms of code, and then force our model to produce more natural original programs written by developers. Learning to generate equivalent, but more natural code, at scale, over large corpora of open-source code, without explicit manual supervision, helps the model learn to both ingest & generate code. We fine-tune our model in three generative Software Engineering tasks: code generation, code translation, and code refinement with limited human-curated labeled data and achieve state-of-the-art performance rivaling CodeT5. We show that our pre-trained model is especially competitive at zero-shot and few-shot learning, and better at learning code properties (e.g., syntax, data flow). CCS Concepts• Software and its engineering → Language features; • Computing methodologies → Knowledge representation and reasoning.

show abstract

Section: Designing Transformation Rulesmentioning

confidence: 99%

NatGen: Generative pre-training by "Naturalizing" source code

Chakraborty¹,

Ahmed²,

Ding³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It was shown that DeepMutation resembles exact matches of 45% of real faulty cases while achieving relatively good syntactic similarity scores in most of the cases. SemSeed [43] aims at inferring faulty patterns from bug-fixes and attempts to generalize them by appropriately adapting them to the particular local code, i.e., context. Although powerful, SemSeed operators on JavaScript programs making its application in our experiment hard.…”

Section: Related Workmentioning

confidence: 99%

“…Although popular, such techniques have been criticised for producing unrealistic faults [12,19,43,49], i.e., faults that are significantly different from real ones in terms of syntax [19], and as a result numerous propositions have been made claiming to produce seeded faults that are syntactically similar to real ones. The most recent research in particular, motivated by the code naturalness hypothesis [21] 1 , aims at forming realistic faults that are, in fact, artificial faults that have some form of syntactic similarity to real ones, i.e., usually following particular syntactic fault patterns.…”

Section: Introductionmentioning

confidence: 99%

“…• We perform the first study on comparing syntactic and semantic similarity of artificial and real faults and found no link between these two metrics. SemSeed: Token Embeddings [43] Derive syntactic patterns that are syntactically similar to real faults Exact syntactic match DeepMutation: Learning-based Mutations [48] Produce mutants syntactically similar to real faults Syntactic distance from real faults Learning-based Mutations [49] Derive syntactic patterns from bug-fixes Syntactic distance from real faults Wild-Caught Mutations [12] Deriving simple syntactic patterns from bug-fixes Token similarity, Compilability Mutation Monkey [11] Deriving common fault syntactic patterns Detection Ratio…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Syntactic Vs. Semantic similarity of Artificial and Real Faults in Mutation Testing Studies

Ojdanić¹,

Garg²,

Khanfir³

et al. 2021

Preprint

View full text Add to dashboard Cite

Fault seeding is typically used in controlled studies to evaluate and compare test techniques. Central to these techniques lies the hypothesis that artificially seeded faults involve some form of realistic properties and thus provide realistic experimental results. In an attempt to strengthen realism, a recent line of research uses advanced machine learning techniques, such as deep learning and Natural Language Processing (NLP), to seed faults that look like (syntactically) real ones, implying that fault realism is related to syntactic similarity. This raises the question of whether seeding syntactically similar faults indeed results in semantically similar faults and more generally whether syntactically dissimilar faults are far away (semantically) from the real ones. We answer this question by employing 4 fault-seeding techniques (PiTest -a popular mutation testing tool, IBIR -a tool with manually crafted fault patterns, DeepMutation -a learning-based fault seeded framework and CodeBERT -a novel mutation testing tool that use code embeddings) and demonstrate that syntactic similarity does not reflect semantic similarity. We also show that 60%, 47%, 43% and 7% of the real faults of Defects4J V2 are semantically resembled by CodeBERT, PiTest, IBIR and DeepMutation faults. We then perform an objective comparison between the techniques and find that CodeBERT and PiTest have similar fault detection capabilities that subsume IBIR and DeepMutation, and that IBIR is the most cost-effective technique. Moreover, the overall fault detection of PiTest, CodeBERT, IBIR and DeepMutation was, on average, 54%, 53%, 37% and 7%.

show abstract

“…RefiNum uses names to identify conceptual types, which further refine the usual programming language types [16]. SemSeed exploits semantic relations between names to inject realistic bugs [42]. All of the above work is based on the observation that the implicit information embedded in identifiers is useful for program analyses.…”

Section: Related Workmentioning

confidence: 99%

Nalin: Learning from Runtime Behavior to Find Name-Value Inconsistencies in Jupyter Notebooks

Patra¹,

Pradel²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Variable names are important to understand and maintain code. If a variable name and the value stored in the variable do not match, then the program suffers from a name-value inconsistency, which is due to one of two situations that developers may want to fix: Either a correct value is referred to through a misleading name, which negatively affects code understandability and maintainability, or the correct name is bound to a wrong value, which may cause unexpected runtime behavior. Finding name-value inconsistencies is hard because it requires an understanding of the meaning of names and knowledge about the values assigned to a variable at runtime. This paper presents Nalin, a technique to automatically detect name-value inconsistencies. The approach combines a dynamic analysis that tracks assignments of values to names with a neural machine learning model that predicts whether a name and a value fit together. To the best of our knowledge, this is the first work to formulate the problem of finding coding issues as a classification problem over names and runtime values. We apply Nalin to 106,652 real-world Python programs, where meaningful names are particularly important due to the absence of statically declared types. Our results show that the classifier detects name-value inconsistencies with high accuracy, that the warnings reported by Nalin have a precision of 80% and a recall of 76% w.r.t. a ground truth created in a user study, and that our approach complements existing techniques for finding coding issues. CCS CONCEPTS• Software and its engineering → Software maintenance tools; Software post-development issues;

show abstract

Semantic bug seeding: a learning-based approach for creating realistic bugs

Cited by 47 publications

References 48 publications

NatGen: Generative pre-training by "Naturalizing" source code

NatGen: Generative pre-training by "Naturalizing" source code

Syntactic Vs. Semantic similarity of Artificial and Real Faults in Mutation Testing Studies

Nalin: Learning from Runtime Behavior to Find Name-Value Inconsistencies in Jupyter Notebooks

Contact Info

Product

Resources

About