Code Defenders: A Mutation Testing Game

Abstract

Abstract-Mutation testing is endorsed by software testing researchers for its unique capability of providing pragmatic estimates of a test suite's fault detection capability, and for guiding testers in improving their test suites. In practice, however, widespread adoption of mutation testing is hampered because any non-trivial program results in huge numbers of mutants, many of which are either trivial or equivalent, and thus useless. Trivial mutants reduce the motivation of developers in trusting and using the technique, while equivalent mutants are frustratingly difficult to handle. These problems are exacerbated by insufficient education on testing, which often means that mutation testing is not well understood in practice. These are examples of the types of problems that gamification aims to overcome by making such tedious activities competitive and entertaining. In this paper, we introduce the first steps towards building CODE DEFENDERS, a mutation testing game where players take the role of an attacker, who aims to create the most subtle non-equivalent mutants, or a defender, who aims to create strong tests to kill these mutants. The benefits of such an approach are manifold: The game can serve an educational role by engaging learners in mutation testing activities in a fun way. Experienced players will produce strong test suites, capable of detecting even the most subtle bugs that other players can conceive. Equivalent mutants are handled by making them a special part of the gameplay, where points are at stake in duels between attackers and defenders.

Gamification in Software Development Projects

“…All participants of research described in [6], [9], [18], [22], and [24] mentioned the best motivation to work, the opportunity to track work progress, better communication among team members, the consolidation of the team and the ability to view their and colleagues' achievements through the use of certain game components, for example, leaderboards, badges or point tables.…”

“…In the event of errors, the functionality is redirected to the development in order to correct errors. Gamification helps one test at a better quality and motivates testers [19], [24].…”

“…The testing phase takes the second place where the majority concerns directly the automatic testing and testing traceability improvement [11], [19], [24]. One article depicted the use of gamification in the PM [10].…”

“…Papers [13]- [28] describe how various tools were developed or studies [5], [24]- [26] provide an insight into the prototypes. Each tool was designed to perform specific tasks: scoring, displaying leaderboards, etc.…”

Towards a Gamified Equivalent Mutants Detection Platform

“…This prevents users from gaming the system by labelling all mutants as equivalent, thus passively gaining points, and creates a competition between users, which was shown to be effective in [12].…”