Crowdsourcing program preconditions via a classification game

Fava, Daniel Schnetzer; Shapiro, David; Osborn, Joseph C.; Schaef, Martin; Whitehead, E. James

doi:10.1145/2884781.2884865

Cited by 7 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fava et al [14] presented an approach to employing GWAP to improve the detection of program invariants. This approach transforms the task of program invariant detection into a computer game, the socalled Binary Fission, which is based on precondition mining.…”

Section: Related Workmentioning

confidence: 99%

Test Data Generation for Program Units Using a Game with a Purpose

2023

View full text Add to dashboard Cite

Today, several methods have been presented to automate test data generation; because of the low maturity level of automatic methods, this is still widely carried out by humans in industry. Hence, the challenge is nding approaches in which humans could generate test data through more attractive, faster, and cheaper ways. To achieve this, one approach is using a Game With A Purpose (GWAP) in test data generation. In our previous work, we introduced two games called Rings and Greenify, by which many inexpensive players with no special technical abilities become engaged in test data generation. This paper presents an entirely new GWAP for test data generation called Quest Of Treasure Explorer (QOTE). QOTE provides a di erent gameplay and has certain advantages compared to prior games, including faster generation of test data, easier puzzles, narration, etc. Experimental results have shown that QOTE outperforms prior games in two aspects: game quality and capability of test data generation. We have also conducted an experiment based on mutation analysis to further evaluate the test data generation capabilities of QOTE compared to four automatic approaches. According to this experiment, QOTE outperforms the four competitors regarding average mutation scores.

show abstract

Section: Related Workmentioning

confidence: 99%

Test Data Generation for Program Units Using a Game with a Purpose

2023

View full text Add to dashboard Cite

show abstract

“…StormBound (Dean et al 2015) presented traces as collections of magic symbols and runes. Binary Fission similarly had players work with invariants, but through a higher-level sorting interface informed by an automated system (Fava et al 2016;Dean et al 2015). Monster Proof (Dean et al 2015) presented information as cartoon monsters.…”

Section: Related Work In Gamesmentioning

confidence: 99%

Gamification of Loop-Invariant Discovery from Code

Walter

Boskin

Cooper

et al. 2019

HCOMP

View full text Add to dashboard Cite

Software verification addresses the important societal problem of software correctness by using tools to mechanically prove that software is free of errors. Since the software verification problem is undecidable, automated tools have limited capabilities; hence, to verify non-trivial software, engineers use human-in-the-loop theorem provers that depend on human-provided insights such as loop invariants. The effective use of modern theorem provers requires significant expertise and recent work has explored the possibility of creating human computation games that enable non-experts to find useful loop invariants. A common feature of these games is that they do not show the code to be verified. We present and evaluate a game which does show players code. Showing code poses a number of design challenges, such as avoiding cognitive overload, but, as our experimental evaluation confirms, also provides an opportunity for richer human-computer interactions that lead to more effective human-in-the-loop systems which augment the ability of programmers who are not verification experts to find loop invariants.

show abstract

“…As discussed in the introduction, several projects investigated the idea of crowdsourcing verification tasks [7,8,12,13,21,23,30]. The main focus of these projects is to investigate to gamification potential and how to create a problem representation that is appealing to gamers.…”

Section: Related Workmentioning

confidence: 99%

“…For example, one of the developed games, GhostMap [23,30], crowdsources the problem of finding suitable refinements in a CEGAR-based model checker. Another game, Xylem [13], crowdsources predicate discovery to assist the abstract interpretation tool Frama-C. Other tools, like BinaryFission [12] crowdsource invariant discovery to obtain more readable invariants than existing machine learning approaches, and Paradox or Pipejam [8] turn constraint systems from type checking into puzzle games.…”

Section: Introductionmentioning

confidence: 99%

Abduction by Non-Experts

Bjørner¹,

Jovanović²,

Lepoint³

et al.

Kalpa Publications in Computing

View full text Add to dashboard Cite

Crowdsourcing promises to quasi-automate tasks that cannot be automated otherwise. Success stories like natural language translation or recognition of cats in images show that carefully crafted crowdsourcing tasks solve large problem instances which could not be solved otherwise. To utilize crowdsourcing, one has to define the problem in a way that is easy to split into small tasks, that the tasks are easy to solve for humans and hard to solve for a machine, and that the machine can efficiently check if the solution is correct.In this paper we discuss a novel approach of using crowdsourcing to assist software verification. We argue that Horn clauses form a good base for crowdsourcing since they are easy to subdivide, and that logic abduction is a suitable task since it is hard to find abductive inferences for Horn clauses automatically, but it is easy to check if an inference makes a Horn clause valid. We describe a prototype implementation, we show how crowdsourcing integrates in the verification process, and present preliminary results.

show abstract

Crowdsourcing program preconditions via a classification game

Cited by 7 publications

References 30 publications

Test Data Generation for Program Units Using a Game with a Purpose

Test Data Generation for Program Units Using a Game with a Purpose

Gamification of Loop-Invariant Discovery from Code

Abduction by Non-Experts

Contact Info

Product

Resources

About