Automatic non-functional testing of code generators families

Abstract

The intensive use of generative programming techniques provides an elegant engineering solution to deal with the heterogeneity of platforms and technological stacks. The use of domain-specific languages for example, leads to the creation of numerous code generators that automatically translate highlevel system specifications into multi-target executable code. Producing correct and efficient code generator is complex and error-prone. Although software designers provide generally high-level test suites to verify the functional outcome of generated code, it remains challenging and tedious to verify the behavior of produced code in terms of non-functional properties. This paper describes a practical approach based on a runtime monitoring infrastructure to automatically check the potential inefficient code generators. This infrastructure, based on system containers as execution platforms, allows code-generator developers to evaluate the generated code performance. We evaluate our approach by analyzing the performance of Haxe, a popular high-level programming language that involves a set of cross-platform code generators. Experimental results show that our approach is able to detect some performance inconsistencies that reveal real issues in Haxe code generators.

Empirical Assessment of Multimorphic Testing

“…In fact, execution times were at least 40 times longer than for morphs targeting other languages. Checking results with authors from previous work [16], [17], they also have noticed this anomaly and reported it to Haxe community in a bug report. Developers responded that they knew about it, it was fixed already but the patch was not live when [16], [17] conducted their study.…”

“…Morphs. Based on previous works presented in [16], [17], we selected 4 popular target languages namely C++, C#, Java, PHP. Then, we tuned code generators according to several optimization parameters they provide.…”

“…Test suites. We reused the same 84 test suites from [16], [17]. Each test suite is composed a number of test cases ranging from 5 to 50.…”

“…Measurements. We used the same testing environment as described in [16], [17], running the same test suites across our 21 morphs, focusing on one property of interest, namely: execution time. We thus collected data relative to the execution time of each generated program.…”

“…We thus collected data relative to the execution time of each generated program. To mitigate the fact that measures could vary because of external factors such as warm-up, or the charge of CPUs, we executed each test case several times on each morph (see [17] for details). Raw measures have been transformed and normalized as follows: (1) Find an execution of reference for each test case and set it to 1.…”