Automated Unit Test Generation for Python

Lukasczyk, Stephan; Kroiß, Florian

doi:10.1007/978-3-030-59762-7_2

Cited by 41 publications

(37 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Without knowing all these details about valid inputs, testing DenseNet using a general-purpose automated test-case generator (for example, Pynguin [16]) would trigger lots of spurious failures, when executing tests that call DenseNet with invalid inputs. The few failing but valid tests that trigger bugs such as that in Lst.…”

Section: An Example Of Using Annotestmentioning

confidence: 99%

“…As we mentioned earlier, NN programs are written in dynamically typed languages like Python, where the type of variables is unknown statically. Therefore, generating valid inputs is challenging with techniques such as random testing and genetic algorithms, which typically assume a variable's type is known [16]. Even if type annotations were available, NN programs routinely manipulate complex data structures-such as vectors, tensors, and other objects-whose precise "shape" is not expressible with the standard types (integers, strings, and so on).…”

Section: Introductionmentioning

confidence: 99%

“…Another confirmed bug we found was due to a function in ADV still using tuple parameter unpacking 15 -a Python 2 feature removed in Python 3. The ADV project developers probably forgot to update this one instance consistently with how they updated the rest of the project 16 , which is indeed designed to work with Python 3.…”

mentioning

confidence: 99%

See 2 more Smart Citations

An Annotation-based Approach for Finding Bugs in Neural Network Programs

Rezaalipour¹,

Furia²

2021

Preprint

View full text Add to dashboard Cite

As neural networks are increasingly included as core components of safety-critical systems, developing effective testing techniques specialized for them becomes crucial. The bulk of the research has focused on testing neural-network models (for instance, their robustness and reliability as classifiers). But neural-network models are defined by writing programs (usually written in a programming language like Python), and there is growing evidence that these neural-network programs often have bugs. Thus, being able to effectively test neural-network programs is instrumental to their dependability.This paper presents aNNoTest: an approach to generating test inputs for neural-network programs. A fundamental challenge is that the dynamically-typed languages (e.g., Python) used to program neural networks cannot express detailed constraints about valid function inputs (e.g., vectors and matrices with certain dimensions). Without knowing these constraints, automated test-case generation is prone to producing many invalid inputs, which trigger spurious failures and are useless for identifying real bugs. To address this problem, we introduce a simple annotation language tailored for expressing valid function inputs in neural-network programs. aNNo-Test inputs an annotated program, and uses property-based testing to generate random inputs that satisfy the validity constraints. In the paper, we also outline guidelines that help reduce the effort needed to write aNNoTest annotations.We evaluated aNNoTest on 19 neural-network programs from Islam et al. 's survey [13], which we manually annotated following our guidelines-producing 6 annotations per tested function on average. aNNoTest automatically generated test inputs that revealed 94 bugs, including 63 bugs that the survey reported for these projects. These results suggest that aNNoTest can be a cost-effective approach to finding widespread bugs in neural-network programs.

show abstract

Section: An Example Of Using Annotestmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Annotation-based Approach for Finding Bugs in Neural Network Programs

Rezaalipour¹,

Furia²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In 2020, Lukasczyk et al introduced a tool, capable of test generation for Python, Pynguin 11 . Pynguin, however, leverages the techniques used in statically typed languages through the assumption that the system under test contains type information with Python's type annotations.…”

Section: Introductionmentioning

confidence: 99%

“…2020 Lukasczyk et al introduced Pynguin11 . A tool that can automatically generate unit tests for Python.…”

mentioning

confidence: 99%

AmPyfier: Test Amplification in Python

Schoofs¹,

Abdi²,

Demeyer³

2021

Preprint

View full text Add to dashboard Cite

Test Amplification is a method to extend handwritten tests into a more rigorous test suite covering corner cases in the system under test. Unfortunately, the current state-of-theart for test amplification heavily relies on program analysis techniques which benefit a lot from explicit type declarations present in statically typed languages like Java and C++. In dynamically typed languages, such type declarations are not available and as a consequence test amplification has yet to find its way to programming languages like Python, Ruby and Javascript. In this paper, we present AmPyfier, a proof-of-concept tool, which brings test amplification to the dynamically typed, interpreted language Python. We evaluated this approach on 7 open-source projects, and found that AmPyfier could successfully strengthen 7 out of 10 test classes (70%). As such we demonstrate that test amplification is feasible for one of the most popular programming languages in use today.

show abstract

AmPyfier: Test amplification in Python

Schoofs

Abdi

Demeyer

2022

J Software Evolu Process

View full text Add to dashboard Cite

Test amplification aims to automatically improve a test suite. One technique generates new test methods through transformations of the original tests. These test amplification tools heavily rely on analysis techniques that benefit a lot from type declarations present in the source code of projects written in statically typed languages. In dynamically typed languages, such type declarations are not available, and therefore, research regarding test amplification for those languages is sparse. Recent work has brought test amplification to the dynamically typed language Pharo Smalltalk by introducing the concept of dynamic type profiling. The technique is dependent on Pharo‐specific frameworks and has not yet been generalized to other languages. Another significant downside in test amplification tools based on the mutation score of a test suite is their high time cost. In this paper, we present AmPyfier, a tool that brings test amplification and type profiling to the dynamically typed language Python. AmPyfier introduces multi‐metric selection in order to increase the time efficiency of test amplification. We evaluated AmPyfier on 11 open‐source projects and found that AmPyfier could strengthen 37 out of 54 test classes. Multi‐metric selection decreased the time cost ranging from 17% to 98% as opposed to selection based on the full mutation score.

show abstract

Automated Unit Test Generation for Python

Cited by 41 publications

References 47 publications

An Annotation-based Approach for Finding Bugs in Neural Network Programs

An Annotation-based Approach for Finding Bugs in Neural Network Programs

AmPyfier: Test Amplification in Python

AmPyfier: Test amplification in Python

Contact Info

Product

Resources

About