RUGRAT: Evaluating program analysis and testing tools and compilers with large generated random benchmark applications

Abstract

Summary Benchmarks are heavily used in different areas of computer science to evaluate algorithms and tools. In program analysis and testing, open‐source and commercial programs are routinely used as benchmarks to evaluate different aspects of algorithms and tools. Unfortunately, many of these programs are written by programmers who introduce different biases, not to mention that it is very difficult to find programs that can serve as benchmarks with high reproducibility of results. We propose a novel approach for generating random benchmarks for evaluating program analysis and testing tools and compilers. Our approach uses stochastic parse trees, where language grammar production rules are assigned probabilities that specify the frequencies with which instantiations of these rules will appear in the generated programs. We implemented our tool for Java and applied it to generate a set of large benchmark programs of up to 5M lines of code each with which we evaluated different program analysis and testing tools and compilers. The generated benchmarks let us independently rediscover several issues in the evaluated tools. Copyright © 2014 John Wiley & Sons, Ltd.

Automatically finding bugs in a commercial cyber-physical system development tool chain with SLforge

“…Empirical studies of widely used programs date back at least to the 1970s [29] and have gained increasing interest due to the wide availability of open source programs [52]. For example, earlier work computed properties from Java programs [12,22,59] and used the properties to guide a random Java program generator [26].…”

Evaluating code‐based test input generator tools

“…The SBST Java Unit Testing Tool Contest invited tool developers to run their tool on several Java classes selected from open source projects, and the tools were ranked based on the coverage and mutation score achieved and the time used. Benchmarks can be created automatically, e.g., RUGRAT is a flexible tool for generating Java programs that can serve as benchmarks for program analysis and testing tools.…”

An improved distributed concolic testing approach

“…Hussain et al [27] proposed random utility generating for program analyzer and testing for generating application benchmarks within the specified constraints and with a range of predefined properties. They have applied stochastic parse trees for generating random application benchmarks.…”