A comparative evaluation on the quality of manual and automatic test case generation techniques for scientific software

Müller, Sebastian; Grunske, Lars

doi:10.1145/3526072.3527523

Cited by 5 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent work, PYNGUIN has also been evaluated on a scientific software stack from material sciences (Trübenbach et al 2022). The authors also compared the performance of different test-generation algorithms and the assertion generation on their subject system; their results regarding the algorithm performance are in line with our results, whereas their results on assertion quality differ significantly due to their use of an older version of PYNGUIN that only provided limited assertion-generation capabilities.…”

Section: Related Worksupporting

confidence: 61%

An empirical study of automated unit test generation for Python

Lukasczyk

Kroiß

2023

Empir Software Eng

View full text Add to dashboard Cite

Various mature automated test generation tools exist for statically typed programming languages such as Java. Automatically generating unit tests for dynamically typed programming languages such as Python, however, is substantially more difficult due to the dynamic nature of these languages as well as the lack of type information. Our Pynguin framework provides automated unit test generation for Python. In this paper, we extend our previous work on Pynguin to support more aspects of the Python language, and by studying a larger variety of well-established state of the art test-generation algorithms, namely DynaMOSA, MIO, and MOSA. Furthermore, we improved our Pynguin tool to generate regression assertions, whose quality we also evaluate. Our experiments confirm that evolutionary algorithms can outperform random test generation also in the context of Python, and similar to the Java world, DynaMOSA yields the highest coverage results. However, our results also demonstrate that there are still fundamental remaining issues, such as inferring type information for code without this information, currently limiting the effectiveness of test generation for Python.

show abstract

Section: Related Worksupporting

confidence: 61%

An empirical study of automated unit test generation for Python

Lukasczyk

Kroiß

2023

Empir Software Eng

View full text Add to dashboard Cite

show abstract

“…The effectiveness of automated test suite implementation strategies for the Python project Atomic Simulation Environment used in the material sciences is examined by Trubenbach et al [1]. The Principal Component Analysis approach by Li et al [2] is suggested in this study for implementing multi-criteria test suites.…”

Section: Literature Reviewmentioning

confidence: 99%

“…One step in the software development life cycle is software artifact testing (SDLC). Some work must be completed as pre-testing operations, such as implementing the test suites [1]. Software testing is necessary to ensure that the software complies with its specifications, but if carried out improperly, it may result in even more problems.…”

Section: Introductionmentioning

confidence: 99%

Implementation of Test Suites using Enhanced State Chart Diagrams: A Case Study

Gupta,

Goyal

2023

J. Sci. Res.

View full text Add to dashboard Cite

An essential part of software engineering is testing. A series of pre-testing tasks should be completed before performing testing tasks. Implementing test suites is one of the pre-testing phases. In this work, a case study has been used to implement the suggested method for implementing test suites. The strategy is based on an analysis of UML (Unified Modelling Language) enhanced state chart diagrams (SCDs). UML state chart diagram analysis, AD (activity diagram) conversion into a graph, AG (activity graph) simplification, and test suite implementation are all steps in the generating process.

show abstract

“…Previously, automated metamorphic relation detection 20 has been used to test NOMAD parsers, resulting in 170 different metamorphic relation candidates. In further related work, the effectiveness of automated test‐case generation approaches was evaluated 21 for the atomic simulation environment, a commonly used Python package for handling atomic structures.…”

Section: Introductionmentioning

confidence: 99%

Grammar‐based fuzzing of data integration parsers in computational materials science

Müller,

Sparka,

Kuban

et al. 2023

Softw Pract Exp

View full text Add to dashboard Cite

ContextComputational materials science (CMS) focuses on in silico experiments to compute the properties of known and novel materials, where many software packages are used in the community. The NOMAD Laboratory (Draxl C, Scheffler) offers to store the input and output files in its FAIR data repository. Since the file formats of these software packages are non‐standardized, parsers are used to provide the results in a normalized format.ObjectiveThe main goal of this article is to report experience and findings of using grammar‐based fuzzing on these parsers.MethodWe have constructed an input grammar for four common software packages in the CMS domain and performed an experimental evaluation on the capabilities of grammar‐based fuzzing to detect failures in the Novel Materials Discovery (NOMAD) parsers.ResultsWith our approach, we were able to identify three unique critical bugs concerning service availability, as well as several additional syntactic, semantic, logical, and downstream bugs in the investigated NOMAD parsers. We reported all issues to the developer team prior to publication.ConclusionBased on the experience gained, we can recommend grammar‐based fuzzing also for other research software packages to improve the trust level in the correctness of the produced results.

show abstract

A comparative evaluation on the quality of manual and automatic test case generation techniques for scientific software

Cited by 5 publications

References 25 publications

An empirical study of automated unit test generation for Python

An empirical study of automated unit test generation for Python

Implementation of Test Suites using Enhanced State Chart Diagrams: A Case Study

Grammar‐based fuzzing of data integration parsers in computational materials science

Contact Info

Product

Resources

About