Using Automated Theorem Provers to Certify Auto-generated Aerospace Software

Abstract

We describe a system for the automated certification of safety properties of NASA software. The system uses Hoare-style program verification technology to generate proof obligations which are then processed by an automated first-order theorem prover (ATP). For full automation, however, the obligations must be aggressively preprocessed and simplified. We discuss the unique requirements this application places on the ATPs and demonstrate how the individual simplification stages, which are implemented by rewriting, influence the ability of the ATPs to solve the proof tasks. Our results are based on 13 certification experiments that lead to more than 25,000 proof tasks which have each been attempted by Vampire, Spass, and e-setheo.

Solving quantified verification conditions using satisfiability modulo theories

“…These cases are safety obligations automatically generated from annotated programs at NASA. Following their introduction in [6], these benchmarks were made publicly available in TPTP format [17], a format for pure first-order logic. We then undertook the task of translating them into the SMT-LIB format and contributing them to the SMT-LIB library.…”

“…We used the following rules to infer types: (i) The index of an array is of type of integer; (ii) The return type of functions cos, sin, log, sqrt is real; (iii) The terms on both sides of infix predicates =, <=, >=, < and >, must have the same type; (iv) If the type of a term cannot be deduced by the above rules, it is assumed to be real. According to [6], of the 28065 cases, only 14 are supposed to be satisfiable (the rest are unsatisfiable). However, after running our experiments and carefully examining the benchmarks in their present form in the TPTP library, our best guess is that somewhere around 150 of the cases are actually satisfiable (both in the SMT-LIB format and in the original TPTP format).…”

“…It is difficult to know for sure since for these cases, no tool we are aware of can reliably distinguish between a truly satisfiable formula and one that is simply too difficult to prove unsatisfiable, and determining this by hand is extremely tedious and error-prone. We suspect that some assumptions present in the benchmarks from [6] were lost somehow before their submission to the TPTP library, but we do not know how this happened. In any case, most of the benchmarks are definitely unsatisfiable and while many are easy, a few of them are very challenging.…”

“…We chose Vampire and SPASS because Vampire and SPASS are among the best ATP systems: Vampire is a regular winner of the CASC competitions [16], and SPASS was the best prover of those tried in [6]. For easier comparison to [6], the benchmarks are divided as in that paper into seven categories: T ∅ , T ∀,→ , T prop , T eval , T array , T policy , T array * . The first category T ∅ contains the most difficult verification conditions.…”

“…We conclude with experimental results demonstrating the effectiveness of our heuristics in improving the performance of CVC3 and in solving verification conditions (in particular, several from the NASA suite introduced in [6]) that no previous ATP or SMT system has been able to solve.…”