A solver-aided language for test input generation

Ringer, Talia; Grossman, Dan; Schwartz-Narbonne, Daniel; Tasiran, Serdar

doi:10.1145/3133915

Cited by 11 publications

(4 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A standard, industry-ready, testing tool, such as EvoSuite [12,[18][19][20], is all one needs to take advantage of June. Ringer et al [38] consider a specification language, Iorek, that provides information to an SMT solver. They use Iorek to generate strings for ATG.…”

Section: Related Workmentioning

confidence: 99%

June: A Type Testability Transformation for Improved ATG Performance

Bruce

Kelly

Menéndez

et al. 2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Strings are universal containers: they are flexible to use, abundant in code, and difficult to test. String-controlled programs are programs that make branching decisions based on string input. Automatically generating valid test inputs for these programs considering only character sequences rather than any underlying string-encoded structures, can be prohibitively expensive.We present June, a tool that enables Java developers to expose any present latent string structure to test generation tools. June is an annotation-driven testability transformation and an extensible library, JuneLib, of structured string definitions. The core JuneLib definitions are empirically derived and provide templates for all structured strings in our test set.June takes lightly annotated source code and injects code that permits an automated test generator (ATG) to focus on the creation of mutable substrings inside a structured string. Using June costs the developer little, with an average of 2.1 annotations per stringcontrolled class. June uses standard Java build tools and therefore deploys seamlessly within a Java project.By feeding string structure information to an ATG tool, June dramatically reduces wasted effort; branches are effortlessly covered that would otherwise be extremely difficult, or impossible, to cover. This waste reduction both increases and speeds coverage. EvoSuite, for example, achieves the same coverage on June-ed classes in 1 minute, on average, as it does in 9 minutes on the un-June-ed class. These gains increase over time. On our corpus, June-ing a program compresses 24 hours of execution time into ca. 2 hours. We show that many ATG tools can reuse the same June-ed code: a few June annotations, a one-off cost, benefit many different testing regimes.

show abstract

Section: Related Workmentioning

confidence: 99%

June: A Type Testability Transformation for Improved ATG Performance

Bruce

Kelly

Menéndez

et al. 2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

show abstract

“…These invariants are passed to a SAT solver that generates satisfying test inputs. Iorek [Ringer et al 2017] allows the programmer to express how inputs differ from each other and then uses an SMT solver to generate different instances of those inputs. Jarvis [Peleg et al 2018] creates property-based tests from a set of preexisting unit tests.…”

Section: Related Workmentioning

confidence: 99%

Programming and execution models for parallel bounded exhaustive testing

Awar

Jain

Rossbach

et al. 2021

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Bounded-exhaustive testing (BET), which exercises a program under test for all inputs up to some bounds, is an effective method for detecting software bugs. Systematic property-based testing is a BET approach where developers write test generation programs that describe properties of test inputs. Hybrid test generation programs offer the most expressive way to write desired properties by freely combining declarative filters and imperative generators. However, exploring hybrid test generation programs, to obtain test inputs, is both computationally demanding and challenging to parallelize. We present the first programming and execution models, dubbed Tempo, for parallel exploration of hybrid test generation programs. We describe two different strategies for mapping the computation to parallel hardware and implement them both for GPUs and CPUs. We evaluated Tempo by generating instances of various data structures commonly used for benchmarking in the BET domain. Additionally, we generated CUDA programs to stress test CUDA compilers, finding four bugs confirmed by the developers.

show abstract

“…In general, beyond Alloy, researchers have focused on improving scenario enumeration strategies, .e.g. dedicated search [3], mixing of generators and solvers [10,14], solver-aided languages [25], and sampling [19,7]. We believe that improvements to the scenarios that get generated by the Analyzer could be refined by some of these approaches, and further combined with Reach.…”

Section: Related Workmentioning

confidence: 99%

REACH: Refining Alloy Scenarios by Scope

Jovanović¹,

Sullivan²

2021

Preprint

View full text Add to dashboard Cite

Writing declarative models has numerous benefits, ranging from automated reasoning and correction of design-level properties before systems are built, to automated testing and debugging of their implementations after they are built. Alloy is a declarative modeling language that is well suited for verifying system designs. A key strength of Alloy is its scenario-finding toolset, the Analyzer, which allows users to explore all valid scenarios that adhere to the model's constraints up to a user-provided scope. In Alloy, it is common for users to desire to first validate smaller scenarios, then once confident, move onto validating larger scenarios. However, the Analyzer only presents scenarios in the order they are discovered by the SAT solver. This paper presents Reach, an extension to the Analyzer which allows users to explore scenarios by size. Experimental results reveal Reach's enumeration improves performance while having the added benefit of maintaining a semi-sorted ordering of scenarios for the user. Moreover, we highlight Reach's ability to improve the performance of Alloy's analysis when the user makes incremental changes to the scope of the enumeration. Reach is available at: https://reachenumerator.github.io/

show abstract

A solver-aided language for test input generation

Cited by 11 publications

References 42 publications

June: A Type Testability Transformation for Improved ATG Performance

June: A Type Testability Transformation for Improved ATG Performance

Programming and execution models for parallel bounded exhaustive testing

REACH: Refining Alloy Scenarios by Scope

Contact Info

Product

Resources

About