Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives

Abstract

Crowdsourcing via paid microtasks has been successfully applied in a plethora of domains and tasks. Previous efforts for making such crowdsourcing more effective have considered aspects as diverse as task and workflow design, spam detection, quality control, and pricing models. Our work expands upon such efforts by examining the potential of adding gamification to microtask interfaces as a means of improving both worker engagement and effectiveness. We run a series of experiments in image labeling, one of the most common use cases for microtask crowdsourcing, and analyse worker behavior in terms of number of images completed, quality of annotations compared against a gold standard, and response to financial and game-specific rewards. Each experiment studies these parameters in two settings: one based on a state-of-the-art, non-gamified task on CrowdFlower and another one using an alternative interface incorporating several game elements. Our findings show that gamification leads to better accuracy and lower costs than conventional approaches that use only monetary incentives. In addition, it seems to make paid microtask work more rewarding and engaging, especially when sociality features are introduced. Following these initial insights, we define a predictive model for estimating the most appropriate incentives for individual workers, based on their previous contributions. This allows us to build a personalised game experience, with gains seen on the volume and quality of work completed.

How to Use Crowdsourcing Effectively

“…There is a large body of research in the crowdsourcing community studying optimal pricing strategies (Archak, 2010;Singer & Mittal, 2013) or the interplay between incentives and other factors such as existing social structures (relevant, for instance, when a crowdsourcing project is carried out in an existing social network or in an enterprise) or motivation. Studies such as (Feyisetan, Simperl, van Kleek, & Shadbolt, 2015;Kaufmann, Schulze, & Veit, 2011) have shown that contributors to crowdsourcing platforms are driven by a rich set of reasons and that relying on financial incentives only might miss great opportunities for engagement and performance gains.…”

Quality Control in Crowdsourcing

“…Designing tasks that induce curiosity is a task-agnostic strategy to improve worker retention [Law et al 2016]. Feyisetan et al [2015] propose a model to predict optimal combinations of financial and gamified incentives.…”

Crowdsourcing planar facility location allocation problems

“…Gamification has also been used in crowdsourcing [39]. In [19], authors have proposed a game called Wordsmith. Wordsmith is a single player game in which players attend in an image labeling task.…”