Gamification of Loop-Invariant Discovery from Code

Abstract

Software verification addresses the important societal problem of software correctness by using tools to mechanically prove that software is free of errors. Since the software verification problem is undecidable, automated tools have limited capabilities; hence, to verify non-trivial software, engineers use human-in-the-loop theorem provers that depend on human-provided insights such as loop invariants. The effective use of modern theorem provers requires significant expertise and recent work has explored the possibility of creating human computation games that enable non-experts to find useful loop invariants. A common feature of these games is that they do not show the code to be verified. We present and evaluate a game which does show players code. Showing code poses a number of design challenges, such as avoiding cognitive overload, but, as our experimental evaluation confirms, also provides an opportunity for richer human-computer interactions that lead to more effective human-in-the-loop systems which augment the ability of programmers who are not verification experts to find loop invariants.

Automated Grading of Automata with ACL2s

“…It comes from the Boyer-Moore family of theorem provers and is capable of reasoning about statements in the first order logic with mathematical induction. ACL2s extends ACL2 with automation and user friendly features like an advanced data definition framework Defdata, a powerful termination analysis based on calling context graphs [20] and ordinals [17,18,19], a property-based modeling and reasoning framework for theorem proving, the cgen framework [8,9,11,31] for generating counter-examples for invalid properties, and support for systems programming, a relatively new capability that allows one to build formal-methods-enabled tools that use ACL2s as a key component and which has been used in projects involving gamified verification, education, proof checking, interfacing with external theorem provers and security [32,29,30].…”