Tests from traces: automated unit test extraction for R

Abstract

Unit tests are labor-intensive to write and maintain. This paper looks into how well unit tests for a target software package can be extracted from the execution traces of client code. Our objective is to reduce the effort involved in creating test suites while minimizing the number and size of individual tests, and maximizing coverage. To evaluate the viability of our approach, we select a challenging target for automated test extraction, namely R, a programming language that is popular for data science applications. The challenges presented by R are its extreme dynamism, coerciveness, and lack of types. This combination decrease the efficacy of traditional test extraction techniques. We present Genthat, a tool developed over the last couple of years to non-invasively record execution traces of R programs and extract unit tests from those traces. We have carried out an evaluation on 1,545 packages comprising 1.7M lines of code. The tests extracted by Genthat improved code coverage from the original rather low value of 267,496 lines to 700,918 lines. The running time of the generated tests is 1.9 times faster than the code they came from.

What we eval in the shadows: a large-scale study of eval in R programs

“…(2) Extract. The genthat tool extracts all runnable code snippets and turns each of these into a selfstanding program [Krikava and Vitek 2018], knitr extracts code from notebooks. Each extracted script is instrumented with calls to our dynamic analyzer in a way that avoids recording calls to eval originating from our execution harness.…”

Model-based testing of breaking changes in Node.js libraries

“…10 This extension is made because NoRegrets+ needs to reconstruct arguments for library functions in the type regression testing phase. 11 We do not need traditional record types or function types, because the different properties of an object or parameters of a function are represented by different paths.…”

“…The SCARPE tool by Joshi and Orso [10] uses a capture phase that generates a model of a software component based on live executions, and a replay phase that can produce regression tests from the model. This is reminiscent of how Krikava and Vitek [11] produce tests for CRAN packages written in R, based on executing the small snippets of executable example code that is often included in the documentation of such packages. In comparison with those techniques, we use the test suites of the client packages to obtain realistic executions of the library.…”

“…https://www.typescriptlang.org/docs/handbook/advanced-types.html11 For readers who are familiar with NoRegrets: NoRegrets+ does not use intersection types nor union types. NoRegrets uses intersection types to model JavaScript prototype chains, however, for NoRegrets+ to reconstruct the client arguments in tests, it must know exactly on which object in a prototype chain a property resides, so extended paths such as p .prototype are used instead to refer to the prototype of a path p. Union types are used by NoRegrets to model polymorphic functions, but are not needed in NoRegrets+ since different calls are distinguished using the κ labels.…”

Formal methods for statistical software

“…For example, the generator may produce the correct result as each case is created, eliminating the need for a separate oracle. One example of a generator is GENTHAT, which generates unit tests from executions of R code [47]. The oracle may be known-good references.…”