A Little Language for Testing

Groce, Alex; Pinto, Jervis

doi:10.1007/978-3-319-17524-9_15

Cited by 17 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More practically, we would like to tightly integrate our approach with automated test generation tools, such as TSTL , DeepState , Echidna and Manticore . Automated test generation tools tend to produce a large number of passing and failing tests and are often used in ‘overnight’ runs where adding a step to use mutants to triage failing tests and provide localization suggestions for the highly ranked tests may not even impose a noticeable overhead on current usage practices.…”

Section: Discussionmentioning

confidence: 99%

Using mutants to help developers distinguish and debug (compiler) faults

Holmes

Groce

2020

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

Summary Measuring the distance between two program executions is a fundamental problem in dynamic analysis of software and useful in many test generation and debugging algorithms. This paper proposes a metric for measuring distance between executions and specializes it to an important application: determining similarity of failing test cases for the purpose of automated fault identification and localization in debugging based on automatically generated compiler tests. The metric is based on a causal concept of distance where executions are similar to the degree that changes in the program itself, introduced by mutation, cause similar changes in the correctness of the executions. Specifically, if two failing test cases for a compiler can be made to pass by applying the same mutant, those two tests are more likely to be due to the same fault. We evaluate our metric using more than 50 faults and 2,800 test cases for two widely used real‐world compilers and demonstrate improvements over state‐of‐the‐art methods for fault identification and localization. A simple operator selection approach to reducing the number of mutants can reduce the cost of our approach by 70%, while producing a gain in fault identification accuracy. We additionally show that our approach, although devised for compilers, is applicable as a conservative fault localization algorithm for other types of programs and can help triage certain types of crashes found in fuzzing non‐compiler programs more effectively than a state‐of‐the‐art technique.

show abstract

Section: Discussionmentioning

confidence: 99%

Using mutants to help developers distinguish and debug (compiler) faults

Holmes

Groce

2020

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

show abstract

“…We present our basic approach in the context of the TSTL [49,58] tool for property-based unit testing of Python programs for several reasons. First, Python is a language with expensive (and coarse-grained: there is no support for path coverage or coverage counts) code coverage tools.…”

Section: Loc-based Heuristicsmentioning

confidence: 99%

Using Relative Lines of Code to Guide Automated Test Generation for Python

Holmes

Ahmed

Brindescu

et al. 2020

ACM Trans. Softw. Eng. Methodol.

Self Cite

View full text Add to dashboard Cite

Raw lines of code (LOC) is a metric that does not, at first glance, seem extremely useful for automated test generation. It is both highly language-dependent and not extremely meaningful, semantically, within a language: one coder can produce the same effect with many fewer lines than another. However, relative LOC , between components of the same project, turns out to be a highly useful metric for automated testing. In this article, we make use of a heuristic based on LOC counts for tested functions to dramatically improve the effectiveness of automated test generation. This approach is particularly valuable in languages where collecting code coverage data to guide testing has a very high overhead. We apply the heuristic to property-based Python testing using the TSTL (Template Scripting Testing Language) tool. In our experiments, the simple LOC heuristic can improve branch and statement coverage by large margins (often more than 20%, up to 40% or more) and improve fault detection by an even larger margin (usually more than 75% and up to 400% or more). The LOC heuristic is also easy to combine with other approaches and is comparable to, and possibly more effective than, two well-established approaches for guiding random testing.

show abstract

“…Among the more "JUnit-like" tools are Pex/IntelliTest [37], UDITA [13], and, in a sense, all model checkers that use the language of the software under test to define the test harness, such as CBMC [23] and Java PathFinder [40]. A related but "inside-out" approach is taken by SPIN [22] (when it is used to model check C code [21]) and by TSTL [17], where the language of the tested system is embedded in a special-purpose language for defining tests and specifications. All of these tools, and DeepState, share the goal of lowering the (often considerable) barriers to the use of automated test generation.…”

Section: Related Workmentioning

confidence: 99%

“…DeepState also targets the same space as property-based testing tools such as QuickCheck [10], ScalaCheck [28], Hypothesis [26], and TSTL [17], [20], but DeepState's test harnesses look like C/C++ unit tests. The major difference from previous tools is that DeepState aims to provide a front-end that can make use of a growing variety of backend methods for test generation, including (already) multiple binary analysis engines and a non-symbolic fuzzer.…”

Section: Introductionmentioning

confidence: 99%

DeepState: Symbolic Unit Testing for C and C++

Goodman¹,

Groce²

2018

Proceedings 2018 Workshop on Binary Analysis Research

Self Cite

View full text Add to dashboard Cite

Unit testing is a popular software development methodology that can help developers detect functional regressions, explore boundary conditions, and document expected behavior. However, writing comprehensive unit tests is challenging and time-consuming, and developers seldom explore the obscure (and bug-hiding) corners of software behavior without assistance. DeepState is a tool that provides a Google Test-like API to give C and C++ developers pushbutton access to symbolic execution engines, such as Manticore and angr, and fuzzers, such as Dr. Fuzz. Rather than learning multiple complex tools, users learn one interface for defining a test harness, and can use various methods to automatically generate tests for software. In addition to providing a familiar interface to binary analysis and fuzzing for parameterized unit testing, DeepState also provides constructs that aid in the construction of API-sequence tests, where the tool chooses the functions or methods to call, allowing for even more diverse and powerful tests. By serving as a front-end to multiple tools, DeepState additionally provides a way to apply (novel) high-level strategies to test generation, and to compare effectiveness and efficiency of testing back-ends, including binary analysis tools.

show abstract

A Little Language for Testing

Cited by 17 publications

References 22 publications

Using mutants to help developers distinguish and debug (compiler) faults

Using mutants to help developers distinguish and debug (compiler) faults

Using Relative Lines of Code to Guide Automated Test Generation for Python

DeepState: Symbolic Unit Testing for C and C++

Contact Info

Product

Resources

About