Testing Static Analyzers with Randomly Generated Programs

Cuoq, Pascal; Monate, Benjamin; Pacalet, Anne; Prévosto, Virgile; Regehr, John; Yakobowski, Boris; Yang, Xuejun

doi:10.1007/978-3-642-28891-3_12

Cited by 60 publications

(25 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This issue has been reported previously as Issue 14 in Randoop's issue tracking system 6 ; i.e., in the test generation phase, if no test is generated after 10 seconds of the last generated test, Randoop terminates without writing any tests, not even the last generated test.…”

Section: Rq3: Rugrat Found Rat Bugs/issuesmentioning

confidence: 88%

See 1 more Smart Citation

Evaluating program analysis and testing tools with the RUGRAT random benchmark application generator

Hussain

Csallner

Grechanik

et al. 2012

Proceedings of the Ninth International Workshop on Dynamic Analysis

View full text Add to dashboard Cite

Benchmarks are heavily used in different areas of computer science to evaluate algorithms and tools. In program analysis and testing, open-source and commercial programs are routinely used as benchmarks to evaluate different aspects of algorithms and tools. Unfortunately, many of these programs are written by programmers who introduce different biases, not to mention that it is very difficult to find programs that can serve as benchmarks with high reproducibility of results.We propose a novel approach for generating random benchmarks for evaluating program analysis and testing tools. Our approach uses stochastic parse trees, where language grammar production rules are assigned probabilities that specify the frequencies with which instantiations of these rules will appear in the generated programs. We implemented our tool for Java and applied it to generate benchmarks with which we evaluated different program analysis and testing tools. Our tool was also implemented by a major software company for C++ and used by a team of developers to generate benchmarks that enabled them to reproduce a bug in less than four hours.

show abstract

Section: Rq3: Rugrat Found Rat Bugs/issuesmentioning

confidence: 88%

“…To achieve the goal, CSmith employs selective construction and analysis of the generated programs. Csmith has been used to test compilers [28] and static analyzers [6]. Unlike RUGRAT, Csmith does not support object-oriented language features.…”

Section: Related Workmentioning

confidence: 99%

Evaluating program analysis and testing tools with the RUGRAT random benchmark application generator

Hussain

Csallner

Grechanik

et al. 2012

Proceedings of the Ninth International Workshop on Dynamic Analysis

View full text Add to dashboard Cite

show abstract

“…The program in Listing 1 is free from undefined and unspecified behaviors, 4,5 although it does involve implementation-defined behavior regarding the representations of data objects. Implementationdefined behaviors are allowed in test cases; indeed, in C, such basic things as the ranges of int and char are implementation-defined.…”

Section: Reporting Compiler Bugsmentioning

confidence: 99%

“…For the sake of clarity, however, we describe the search for i Bmin as if it were unique. 4 The C99 language standard [7] identifies many program behaviors that are undefined or unspecified. An example undefined behavior is dereferencing a null pointer.…”

Section: The Test-case Minimization Problemmentioning

confidence: 99%

See 1 more Smart Citation

Test-case reduction for C compiler bugs

et al. 2012

Self Cite

View full text Add to dashboard Cite

To report a compiler bug, one must often find a small test case that triggers the bug. The existing approach to automated test-case reduction, delta debugging, works by removing substrings of the original input; the result is a concatenation of substrings that delta cannot remove. We have found this approach less than ideal for reducing C programs because it typically yields test cases that are too large or even invalid (relying on undefined behavior). To obtain small and valid test cases consistently, we designed and implemented three new, domain-specific test-case reducers. The best of these is based on a novel framework in which a generic fixpoint computation invokes modular transformations that perform reduction operations. This reducer produces outputs that are, on average, more than 25 times smaller than those produced by our other reducers or by the existing reducer that is most commonly used by compiler developers. We conclude that effective program reduction requires more than straightforward delta debugging.

show abstract

RUGRAT: Evaluating program analysis and testing tools and compilers with large generated random benchmark applications

Hussain

Csallner

Grechanik

et al. 2014

Softw. Pract. Exper.

View full text Add to dashboard Cite

Summary Benchmarks are heavily used in different areas of computer science to evaluate algorithms and tools. In program analysis and testing, open‐source and commercial programs are routinely used as benchmarks to evaluate different aspects of algorithms and tools. Unfortunately, many of these programs are written by programmers who introduce different biases, not to mention that it is very difficult to find programs that can serve as benchmarks with high reproducibility of results. We propose a novel approach for generating random benchmarks for evaluating program analysis and testing tools and compilers. Our approach uses stochastic parse trees, where language grammar production rules are assigned probabilities that specify the frequencies with which instantiations of these rules will appear in the generated programs. We implemented our tool for Java and applied it to generate a set of large benchmark programs of up to 5M lines of code each with which we evaluated different program analysis and testing tools and compilers. The generated benchmarks let us independently rediscover several issues in the evaluated tools. Copyright © 2014 John Wiley & Sons, Ltd.

show abstract

Testing Static Analyzers with Randomly Generated Programs

Cited by 60 publications

References 2 publications

Evaluating program analysis and testing tools with the RUGRAT random benchmark application generator

Evaluating program analysis and testing tools with the RUGRAT random benchmark application generator

Test-case reduction for C compiler bugs

RUGRAT: Evaluating program analysis and testing tools and compilers with large generated random benchmark applications

Contact Info

Product

Resources

About