Massive Multiplayer Human Computation for Fun, Money, and Survival

Abstract

Crowdsourcing is an effective tool to solve hard tasks. By bringing 100,000s of people to work on simple tasks that only humans can do, we can go far beyond traditional models of data analysis and machine learning. As technologies and processes mature, crowdsourcing is becoming mainstream. It powers many leading Internet companies and a wide variety of novel projects: from content moderation and business listing verification to real-time SMS translation for disaster response. However, quality assurance can be a major challenge. In this paper CrowdFlower presents various crowdsourcing applications, from business to ethics, to money and survival, all of which showcase the power of labor-on-demand, otherwise known as the human cloud.

An innovative web-based collaborative platform for video annotation

“…The easiest and most natural way to motivate people is paying them for their work. This strategy is applied by Amazon's Mechanical Turk service [32] and CrowdFlower [7]. A valid alternative for workers motivation is personal amusement [43]: this is the case of the ESP and Peekaboom games [1] which exploit players' agreement (randomly pairing two players and let them guess each other's labels) to collect ground truth data.…”

Generating reliable video annotations by exploiting the crowd

“…The Amazon's Mechanical Turk service [31] and CrowdFlower [10] have been the first crowdsourcing implementations, which revolutionised static data annotation in computer vision and enabled many large scale image datasets to be labeled [12,33,22]. These systems, basically, enable people to pay to get annotations by crowd workers and require specific techniques to verify and control over crowdsourced annotations [36,41,30,35].…”

“…Despite the existence of several tools to support users in the labeling task [15,5,18], in recent years much research has been devoted to find methods for automatic or semi-automatic ground truth generation [33,2,1,17,34]. Besides semi-automatic [19] and web-based collaborative solutions [20], crowdsourcing to non-experts has recently been adopted with the objective of supporting the generation of high-quality annotated datasets [41,31,10].…”

Implementation and Analytics of the Distributed Eco-Driving Simulation iCO2

“…Biewald [7] presents a number of applications involving crowdsourcing and human computation that use the increasingly popular Mechanical Turk from Amazon [30] and involve ethics, business and fun. The tasks involve answering questions that do not require any technical knowledge or performing Google searches.…”