Systematically Covering Input Structure

Havrikov, Nikolas; Zeller, Andreas

doi:10.1109/ase.2019.00027

Cited by 31 publications

(11 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, our approach can be extended to learn contextual dependencies, e.g. by learning sequences of elements using N-grams [15] or hierarchies of elements using k-paths [16]. Input Constraints: Beyond lexical and syntactical validity, structured inputs often contain input semantics such as checksums, hashes, encryption, or references.…”

Section: Limitationsmentioning

confidence: 99%

Inputs From Hell:

Soremekun

Pavese

Havrikov

et al. 2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Grammars can serve as producers for structured test inputs that are syntactically correct by construction. A probabilistic grammar assigns probabilities to individual productions, thus controlling the distribution of input elements. Using the grammars as input parsers, we show how to learn input distributions from input samples, allowing to create inputs that are similar to the sample; by inverting the probabilities, we can create inputs that are dissimilar to the sample. This allows for three test generation strategies: 1) "Common inputs" -by learning from common inputs, we can create inputs that are similar to the sample; this is useful for regression testing. 2) "Uncommon inputs" -learning from common inputs and inverting probabilities yields inputs that are strongly dissimilar to the sample; this is useful for completing a test suite with "inputs from hell" that test uncommon features, yet are syntactically valid. 3) "Failure-inducing inputs" -learning from inputs that caused failures in the past gives us inputs that share similar features and thus also have a high chance of triggering bugs; this is useful for testing the completeness of fixes. Our evaluation on three common input formats (JSON, JavaScript, CSS) shows the effectiveness of these approaches. Results show that "common inputs" reproduced 96% of the methods induced by the samples. In contrast, for almost all subjects (95%), the "uncommon inputs" covered significantly different methods from the samples. Learning from failure-inducing samples reproduced all exceptions (100%) triggered by the failure-inducing samples and discovered new exceptions not found in any of the samples learned from.

show abstract

Section: Limitationsmentioning

confidence: 99%

Inputs From Hell:

Soremekun

Pavese

Havrikov

et al. 2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Coverage-guided fuzzing tools like AFL [22] can be viewed as a way of using a different form of feedback (branch instead of combinatorial coverage) to improve the generation of random inputs by finding more "interesting" tests. Fuzzing is a huge topic [43] that has exploded in popularity recently, with researchers evaluating the benefits of using more forms of feedback [13,31], incorporating learning [28,33] or symbolic [39,42] techniques, and bringing the benefits of these methods to functional programming [11,21]. One fundamental difference, however, is that all of these techniques are online and grey-box: they instrument and execute the program on various inputs in order to obtain feedback.…”

Section: Comparison With Fuzzing Techniquesmentioning

confidence: 99%

Do Judge a Test by its Cover

Goldstein

Hughes

Lampropoulos

et al. 2021

Programming Languages and Systems

View full text Add to dashboard Cite

Property-based testing uses randomly generated inputs to validate high-level program specifications. It can be shockingly effective at finding bugs, but it often requires generating a very large number of inputs to do so. In this paper, we apply ideas from combinatorial testing, a powerful and widely studied testing methodology, to modify the distributions of our random generators so as to find bugs with fewer tests. The key concept is combinatorial coverage, which measures the degree to which a given set of tests exercises every possible choice of values for every small combination of input features.In its “classical” form, combinatorial coverage only applies to programs whose inputs have a very particular shape—essentially, a Cartesian product of finite sets. We generalize combinatorial coverage to the richer world of algebraic data types by formalizing a class of sparse test descriptions based on regular tree expressions. This new definition of coverage inspires a novel combinatorial thinning algorithm for improving the coverage of random test generators, requiring many fewer tests to catch bugs. We evaluate this algorithm on two case studies, a typed evaluator for System F terms and a Haskell compiler, showing significant improvements in both.

show abstract

“…2) Input Structure: Since we know the ChocoPy syntax, we can consider systematically enumerating k-paths [22] within the ChocoPy grammar. This approach yields minimal programs corresponding to each unique k-length path (from root to leaf) in valid syntax trees.…”

Section: A Systematic Testingmentioning

confidence: 99%

Growing a Test Corpus with Bonsai Fuzzing

Vikram

Padhye

Sen

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

This paper presents a coverage-guided grammarbased fuzzing technique for automatically synthesizing a corpus of concise test inputs. We walk-through a case study of a compiler designed for education and the corresponding problem of generating meaningful test cases to provide to students. The prior state-of-the-art solution is a combination of fuzzing and test-case reduction techniques such as variants of deltadebugging. Our key insight is that instead of attempting to minimize convoluted fuzzer-generated test inputs, we can instead grow concise test inputs by construction using a form of iterative deepening. We call this approach bonsai fuzzing. Experimental results show that bonsai fuzzing can generate test corpora having inputs that are 16-45% smaller in size on average as compared to a fuzz-then-reduce approach, while achieving approximately the same code coverage and fault-detection capability.

show abstract

Systematically Covering Input Structure

Cited by 31 publications

References 19 publications

Inputs From Hell:

Inputs From Hell:

Do Judge a Test by its Cover

Growing a Test Corpus with Bonsai Fuzzing

Contact Info

Product

Resources

About