Combinatorial Interaction Testing for Test Selection in Grammar-Based Testing

Salecker, Elke; Glesner, Sabine

doi:10.1109/icst.2012.148

Cited by 7 publications

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Do Judge a Test by its Cover

Goldstein

Hughes

Lampropoulos

et al. 2021

Programming Languages and Systems

View full text Add to dashboard Cite

Property-based testing uses randomly generated inputs to validate high-level program specifications. It can be shockingly effective at finding bugs, but it often requires generating a very large number of inputs to do so. In this paper, we apply ideas from combinatorial testing, a powerful and widely studied testing methodology, to modify the distributions of our random generators so as to find bugs with fewer tests. The key concept is combinatorial coverage, which measures the degree to which a given set of tests exercises every possible choice of values for every small combination of input features.In its “classical” form, combinatorial coverage only applies to programs whose inputs have a very particular shape—essentially, a Cartesian product of finite sets. We generalize combinatorial coverage to the richer world of algebraic data types by formalizing a class of sparse test descriptions based on regular tree expressions. This new definition of coverage inspires a novel combinatorial thinning algorithm for improving the coverage of random test generators, requiring many fewer tests to catch bugs. We evaluate this algorithm on two case studies, a typed evaluator for System F terms and a Haskell compiler, showing significant improvements in both.

show abstract

Do Judge a Test by its Cover

Goldstein

Hughes

Lampropoulos

et al. 2021

Programming Languages and Systems

View full text Add to dashboard Cite

show abstract

Enhancing Synthetic Test Data Generation with Language Models Using a More Expressive Domain-Specific Language

Tan,

Behjati,

Arisholm

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Application of deep learning models to generate rich, dynamic and production-like test data

Tan,

Behjati,

Arisholm

2024

Empir Software Eng

View full text Add to dashboard Cite

Traditionally, software development teams in many industries have used copies of production databases or their masked, anonymized, or obfuscated versions for testing. However, privacy protection regulations, for example, the General Data Protection Regulation (GDPR), prohibit such practices. In such a situation, there is often a need to generate production-like test data, i.e., test data that is statistically representative of the production data and conforms to the domain’s constraints. In this paper, we address this need by presenting a novel approach for generating production-like test data using deep learning techniques and studying the practical effectiveness of our proposed approach in industrial settings. We frame the problem of generating production-like test data as a Language Modeling problem. We then propose a general solution for test data generation and a framework for evaluating and comparing language models based on training effectiveness and the representativeness and validity of the generated data. To evaluate the practical effectiveness of our solution, we apply it to a case study: the Norwegian National Population Registry (NPR). Within the context of NPR, we experiment with three of the most successful Deep Learning algorithms for Language Modeling, namely Recurrent Neural Networks (RNN), Variational Autoencoders, and Generative Adversarial Networks (GANs). We furthermore evaluate and compare the effectiveness of these algorithms quantitatively, using the proposed evaluation framework. The results from our case study show that our approach can generate highly complex data that is statistically representative of the production data and conforms to the business rules of the domain. Moreover, test data generated with RNN, which outperforms the other two algorithms, are syntactically and semantically valid in more than 97% of the cases and are highly representative of the real NPR data. The practical applicability of our approach is evident from the fact that our approach was fully deployed in a test environment at the NPR, generating on-the-fly and scalable amounts of production-like test data that are used in the integration testing between NPR and its data consumers.

show abstract

Combinatorial Interaction Testing for Test Selection in Grammar-Based Testing

Cited by 7 publications

References 13 publications

Do Judge a Test by its Cover

Do Judge a Test by its Cover

Enhancing Synthetic Test Data Generation with Language Models Using a More Expressive Domain-Specific Language

Application of deep learning models to generate rich, dynamic and production-like test data

Contact Info

Product

Resources

About