Making the non-executable ACATS tests executable

Abstract

The Ada Conformity Assessment Test Suite (ACATS) includes both positive and negative tests. The negative tests have intentional errors that a compiler is intended to diagnose with a compile-time error message. Interestingly, the negative tests also include numerous "OK" lines that the compiler must not reject. But the absence of an error message on the "OK" lines is not always a convincing demonstration that the "OK" lines were correctly compiled, since the negative tests are not executable. By removing the intentional errors from a copy of each negative test, we are able to compile and run the resulting modified tests, demonstrating more convincingly that the "OK" lines are correctly compiled.

Adapting ACATS for use with run-time checks suppressed

“…We have proposed a means of removing the intentional errors from a copy of each of the non-executable tests so that the remaining code, which includes these numerous positive test objectives, can be tested in the context of an executable test. Testing an executable feature in an executable test gives much higher confidence that it is implemented correctly than if it is tested only in a non-executable test [6].…”

“…For example, one might notice that numeric overflow checking is tested for integer types and floating point types, but not for fixed point types, as pointed out in a previous paper [6].…”