Augmenting Field Data for Testing Systems Subject to Incremental Requirements Changes

Abstract

When testing data processing systems, software engineers often use real world data to perform system level testing. However, in the presence of new data requirements software engineers may no longer benefit from having real world data with which to perform testing. Typically, new test inputs complying with the new requirements have to be manually written.We propose an automated model-based approach that combines data modelling and constraint solving to modify existing field data to generate test inputs for testing new data requirements. The approach scales in the presence of complex and structured data, thanks to both the reuse of existing field data and the adoption of an innovative input generation algorithm based on slicing the model into parts.We validated the scalability and effectiveness of the proposed approach using an industrial case study. The empirical study shows that the approach scales in the presence of large amounts of structured and complex data. The approach can produce, within a reasonable time, test input data that is over ten times larger in size than the data generated with constraint solving only. We also demonstrate that the generated test inputs achieve more code coverage than the test cases implemented by experienced software engineers.

Practical Constraint Solving for Generating System Test Data

“…Our case study material is borrowed from previous work. Specifically, Case A was built for system testing of an eGovernment application concerned with calculating personal income taxes [77], Case B for system testing of an occupant detection system in cars [82], and Case C for system testing of a satellite communication system [28]. Our case study material (in a sanitized form) is available from PLEDGE's website.…”

“…expand is called recursively over original's children */ originalshortcuts alongside their equivalent expressions with explicit quantification. In the final segment of the algorithm (L [28][29][30][31],. expansion is applied recursively to the children of a visited node.…”

Human-based Test Design versus Automated Test Generation: A Literature Review and Meta-Analysis

“…The authors compare the effectiveness of defects found and efficiency in terms of effort spent to complete each process. P8 [9]: In this paper, the authors propose an automated modelbased approach to generate test inputs for testing new data requirements by modifying the old data. P9: [30]: In this paper, the authors design and conduct an empirical experiment to compare manual ad hoc testing with model-based testing with the aid of the TaRGeT tool.…”

Multiple conformance to hybrid-automata-modelled requirements for detecting indoor temperature anomalies

“…For example, the specification has to be extended to deal with new situations. Di Nardo et al 51 presented an automated model-based approach that combines data modelling and constraint solving to modify existing field data to generate test inputs for testing new data requirements. A toolset for automatic conformance testing is presented in Faria and Paiva 52 that enable incremental conformance checking.…”