Back-To-Back Testing of Model-Based Code Generators

Jörges, Sven; Steffen, Bernhard

doi:10.1007/978-3-662-45234-9_30

Cited by 8 publications

(9 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They use Simulink as a simulation environment of models. In [54], authors present a testing approach of the Genesys code generator framework which tests the translation performed by a code generator from a semantic perspective rather that just checking for syntactic correctness of the generation result. Basically, Genesys realizes back-to-back testing by executing both the source model as well as the generated code on top of different target platforms.…”

Section: Testing and Code Generatorsmentioning

confidence: 99%

Empirical Assessment of Multimorphic Testing

Temple

Acher

Jézéquel

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

The performance of software systems such as speed, memory usage, correct identification rate, tends to be an evermore important concern, often nowadays on par with functional correctness for critical systems. Systematically testing these performance concerns is however extremely difficult, in particular because there exists no theory underpinning the evaluation of a performance test suite, i.e., to tell the software developer whether such a test suite is "good enough" or even whether a test suite is better than another one. This paper proposes to apply Multimorphic testing and empirically assess the effectiveness of performance test suites of software systems coming from various domains. By analogy with mutation testing, our core idea is to leverage the typical configurability of these systems, and to check whether it makes any difference in the outcome of the tests: i.e., are some tests able to "kill" underperforming system configurations? More precisely, we propose a framework for defining and evaluating the coverage of a test suite with respect to a quantitative property of interest. Such properties can be the execution time, the memory usage or the success rate in tasks performed by a software system. This framework can be used to assess whether a new test case is worth adding to a test suite or to select an optimal test suite with respect to a property of interest. We evaluate several aspects of our proposal through 3 empirical studies carried out in different fields: object tracking in videos, object recognition in images, and code generators.

show abstract

Section: Testing and Code Generatorsmentioning

confidence: 99%

Empirical Assessment of Multimorphic Testing

Temple

Acher

Jézéquel

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Most of the previous work on code generator testing focuses on checking the correct functional behaviour of generated code .Most of these research efforts rely on the comparison of the model execution to the generated code execution. This is known in the software testing community as equivalence, comparative or back‐to‐back testing approach .…”

Section: Related Workmentioning

confidence: 99%

Leveraging metamorphic testing to automatically detect inconsistencies in code generator families

Boussaa

Barais

Sunyé

et al. 2019

Software Testing Verif & Rel

View full text Add to dashboard Cite

SUMMARY Generative software development has paved the way for the creation of multiple code generators that serve as a basis for automatically generating code to different software and hardware platforms. In this context, the software quality becomes highly correlated to the quality of code generators used during software development. Eventual failures may result in a loss of confidence for the developers, who will unlikely continue to use these generators. It is then crucial to verify the correct behaviour of code generators in order to preserve software quality and reliability. In this paper, we leverage the metamorphic testing approach to automatically detect inconsistencies in code generators via so‐called “metamorphic relations”. We define the metamorphic relation (i.e., test oracle) as a comparison between the variations of performance and resource usage of test suites running on different versions of generated code. We rely on statistical methods to find the threshold value from which an unexpected variation is detected. We evaluate our approach by testing a family of code generators with respect to resource usage and performance metrics for five different target software platforms. The experimental results show that our approach is able to detect, among 95 executed test suites, 11 performance and 15 memory usage inconsistencies.

show abstract

“…A reliable and acceptable way to increase the confidence in the correctness of a code generator family is to validate and check the functionality of generated code, which is a common practice for compiler validation and testing [11,25,26]. Therefore, developers try to check the syntactic and semantic correctness of the generated code by means of different techniques such as static analysis, test suites, etc., and ensure that the code is behaving correctly.…”

Section: Functional Correctness Of a Code Generator Familymentioning

confidence: 99%

“…We use InfluxDB 11 , an open source distributed time-series database as a back-end to record data. InfluxDB allows the user to execute SQL-like queries on the database.…”

Section: Back-end Database Componentmentioning

confidence: 99%

“…They have used Simulink as a simulation environment of models. In [11], authors presented a testing approach of the Genesys code generator framework which tests the translation performed by a code generator from a semantic perspective rather than just checking for syntactic correctness of the generation result. Basically, Genesys realizes back-to-back testing by executing both the source model as well as the generated code on top of different target platforms.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Automatic non-functional testing of code generators families

et al. 2016

View full text Add to dashboard Cite

The intensive use of generative programming techniques provides an elegant engineering solution to deal with the heterogeneity of platforms and technological stacks. The use of domain-specific languages for example, leads to the creation of numerous code generators that automatically translate highlevel system specifications into multi-target executable code. Producing correct and efficient code generator is complex and error-prone. Although software designers provide generally high-level test suites to verify the functional outcome of generated code, it remains challenging and tedious to verify the behavior of produced code in terms of non-functional properties. This paper describes a practical approach based on a runtime monitoring infrastructure to automatically check the potential inefficient code generators. This infrastructure, based on system containers as execution platforms, allows code-generator developers to evaluate the generated code performance. We evaluate our approach by analyzing the performance of Haxe, a popular high-level programming language that involves a set of cross-platform code generators. Experimental results show that our approach is able to detect some performance inconsistencies that reveal real issues in Haxe code generators.

show abstract

Back-To-Back Testing of Model-Based Code Generators

Cited by 8 publications

References 40 publications

Empirical Assessment of Multimorphic Testing

Empirical Assessment of Multimorphic Testing

Leveraging metamorphic testing to automatically detect inconsistencies in code generator families

Automatic non-functional testing of code generators families

Contact Info

Product

Resources

About