Small-Amp: Test amplification in a dynamically typed language

Abstract

Some test amplification tools extend a manually created test suite with additional test cases to increase the code coverage. The technique is effective, in the sense that it suggests strong and understandable test cases, generally adopted by software engineers. Unfortunately, the current state-of-the-art for test amplification heavily relies on program analysis techniques which benefit a lot from explicit type declarations present in statically typed languages. In dynamically typed languages, such type declarations are not available and as a consequence test amplification has yet to find its way to programming languages like Smalltalk, Python, Ruby and Javascript. We propose to exploit profiling information —readily obtainable by executing the associated test suite— to infer the necessary type information creating special test inputs with corresponding assertions. We evaluated this approach on 52 selected test classes from 13 mature projects in the Pharo ecosystem containing approximately 400 test methods. We show the improvement in killing new mutants and mutation coverage at least in 28 out of 52 test classes (≈ 53%). Moreover, these generated tests are understandable by humans: 8 out of 11 pull-requests submitted were merged into the main code base (≈ 72%). These results are comparable to the state-of-the-art, hence we conclude that test amplification is feasible for dynamically typed languages.

An empirical study of automated unit test generation for Python

“…Schoofs et al (2022) introduce PYAMPLIFIER a proof-of-concept tool for test amplification in Python. Recently, SMALL-AMP (Abdi et al 2022) demonstrated the use of dynamic type profiling as a substitute for type declarations in the Pharo Smalltalk language.…”

AmPyfier: Test amplification in Python

“…Tools belonging to AMP add will generate new test methods based on the original handwritten test methods. The newly generated test methods try to improve the test suite based on metrics such as mutation score (e.g., DSpot and Small-Amp), 7,9 code coverage, [15][16][17] and reproduction of crashes (e.g., MuCrash). 18,19 AMP add is a special case of AMP change , as its tools and techniques only focus on the changed parts of the project under test compared to its previous version (e.g., DCI).…”

“…Dynamic type profiling is introduced by Small‐Amp for test amplification in the dynamically typed language Pharo Smalltalk 9 by exploiting a Pharo‐specific reflectivity mechanism called Metalinks. However, the Metalinks implementation in Pharo takes advantage of the live nature of Pharo.…”

“…Dynamic techniques are needed to deduce the type of variables and values that are to be asserted and to support type‐sensitive input amplifiers. Abdi et al 8,9 have made a tool, Small‐Amp with which they showed that test amplification is also possible for the dynamically typed language Pharo Smalltalk. They adopted the concept of dynamic type profiling, with which they were able to overcome the lack of type information in the source code.…”

Test Transplantation through Dynamic Test Slicing

“…Exploiting this source of knowledge to improve software testing is called test amplification [3]. SMALL-AMP [4] is a test amplifier in Pharo that synthesizes new unit tests that increase the mutation coverage. SMALL-AMP is based on four main components: (1) a profiler which captures variables' type information, (2) an input amplifier which transforms existing test methods and generates new test inputs, (3) an assertion amplifier which regenerates the assertion statements, and (4) a selector that runs mutation testing and identifies the tests introducing new coverage.…”