Inferring Loop Invariants through Gamification

Bounov, Dimitar; DeRossi, Anthony; Menarini, Massimiliano; Griswold, William G.; Lerner, Sorin

doi:10.1145/3173574.3173805

Cited by 14 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2018, Bounov et al developed a loop invariant discovery game in which, unlike many of the CSFV games, users are exposed to mathematical symbols and operators directly [7]. This game, called InvGame, is similar to many of the CSFV games in that it does not display the code being reasoned about.…”

Section: Related Workmentioning

confidence: 99%

“…We note that the characterization problem is undecidable, as we have to check the validity of formulas that include addition and multiplication, hence, our final characterization. (7) unknown: When none of the above cases hold, we say that e is unknown.…”

Section: Expression Characterizationsmentioning

confidence: 99%

“…The major contribution of this paper is the design of a reasoning engine that allows students and programmers without formal methods knowledge to collaboratively prove statements about programs which were not provable by best-in-class automated tools [7], with minimal training, in under one hour. We describe the core algorithms for taking incomplete proposed invariants from multiple sources and combining them to verify programs and to produce concrete and actionable feedback enabling users to propose further invariants, without having to understand formal proofs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Reasoning Engine for the Gamification of Loop-Invariant Discovery

Walter¹,

Cooper²,

Manolios³

2021

Preprint

View full text Add to dashboard Cite

We describe the design and implementation of a reasoning engine that facilitates the gamification of loop-invariant discovery. Our reasoning engine enables students, computational agents and regular software engineers with no formal methods expertise to collaboratively prove interesting theorems about simple programs using browserbased, online games. Within an hour, players are able to specify and verify properties of programs that are beyond the capabilities of fullyautomated tools. The hour limit includes the time for setting up the system, completing a short tutorial explaining game play and reasoning about simple imperative programs. Players are never required to understand formal proofs; they only provide insights by proposing invariants. The reasoning engine is responsible for managing and evaluating the proposed invariants, as well as generating actionable feedback.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Expression Characterizationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Reasoning Engine for the Gamification of Loop-Invariant Discovery

Walter¹,

Cooper²,

Manolios³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In this section we empirically evaluate PLearn and HEnum. Our evaluation uses a set of 180 synthesis benchmarks, 6 consisting of all 127 official benchmarks from the Inv track of 2018 SyGuS competition [4] augmented with benchmarks from the 2018 Software Verification competition (SV-Comp) [8] and challenging verification problems proposed in prior work [9,10]. All these synthesis tasks are defined over integer and Boolean values, and we evaluate them with the six grammars described in Fig.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Overfitting in Synthesis: Theory and Practice

Padhi

Millstein

Nori

et al. 2019

Computer Aided Verification

View full text Add to dashboard Cite

In syntax-guided synthesis (SyGuS), a synthesizer's goal is to automatically generate a program belonging to a grammar of possible implementations that meets a logical specification. We investigate a common limitation across state-of-the-art SyGuS tools that perform counterexample-guided inductive synthesis (CEGIS). We empirically observe that as the expressiveness of the provided grammar increases, the performance of these tools degrades significantly.We claim that this degradation is not only due to a larger search space, but also due to overfitting. We formally define this phenomenon and prove no-free-lunch theorems for SyGuS, which reveal a fundamental tradeoff between synthesizer performance and grammar expressiveness.A standard approach to mitigate overfitting in machine learning is to run multiple learners with varying expressiveness in parallel. We demonstrate that this insight can immediately benefit existing SyGuS tools. We also propose a novel single-threaded technique called hybrid enumeration that interleaves different grammars and outperforms the winner of the SyGuS competition (Inv track), solving more problems and achieving a 5× mean speedup. *

show abstract

“…There is some work on gamification of proofs that reifies this intuition into the interface itself. In these games, players can generate program annotations (Dietl et al, 2012), write natural deduction proofs (Lerner et al, 2015), and identify inductive invariants (Bounov et al, 2018), all the while having low-level details of these proofs abstracted away from them. While these games are not interfaces for well-known ITPs like Coq and Isabelle, they may help with tasks that can assist users in writing proofs, such as finding inductive invariants.…”

Section: Specialized Interfacesmentioning

confidence: 99%

QED at Large: A Survey of Engineering of Formally Verified Software

Ringer

Palmskog

Sergey

et al. 2019

FNT in Programming Languages

View full text Add to dashboard Cite

Development of formal proofs of correctness of programs can increase actual and perceived reliability and facilitate better understanding of program specifications and their underlying assumptions. Tools supporting such development have been available for over 40 years, but have only recently seen wide practical use. Projects based on construction of machine-checked formal proofs are now reaching an unprecedented scale, comparable to large software projects, which leads to new challenges in proof development and maintenance. Despite its increasing importance, the field of proof engineering is seldom considered in its own right; related theories, techniques, and tools span many fields and venues. This survey of the literature presents a holistic understanding of proof engineering for program correctness, covering impact in practice, foundations, proof automation, proof organization, and practical proof development.

show abstract

Inferring Loop Invariants through Gamification

Cited by 14 publications

References 25 publications

A Reasoning Engine for the Gamification of Loop-Invariant Discovery

A Reasoning Engine for the Gamification of Loop-Invariant Discovery

Overfitting in Synthesis: Theory and Practice

QED at Large: A Survey of Engineering of Formally Verified Software

Contact Info

Product

Resources

About