Combining crowdsourcing and learning to improve engagement and performance

Abstract

Crowdsourcing complex creative tasks remains difficult, in part because these tasks require skilled workers. Most crowdsourcing platforms do not help workers acquire the skills nec essary to accomplish complex creative tasks. In this paper, we describe a platform that combines learning and crowdsourc ing to benefit both the workers and the requesters. Workers gain new skills through interactive step-by-step tutorials and test their knowledge by improving real-world images submit ted by requesters. In a series of three deployments spanning two years, we varied the design of our platform to enhance the learning experience and improve the quality of the crowd work. We tested our approach in the context of LevelUp for Photoshop, which teaches people how to do basic photograph improvement tasks using Adobe Photoshop. We found that by using our system workers gained new skills and produced high-quality edits for requested images, even if they had little prior experience editing images.

Quality Control in Crowdsourcing

“…Share purpose. Tasks that have a purpose that goes beyond the individual micro-task, that workers understand and can identify with can help attract crowds to perform tasks with higher motivation, also for free by volunteering [Dontcheva et al 2014]. Examples of crowdsourcing initiatives driven by this sense of purpose are Zoouniverse (www.zooniverse.org) described in [Mao et al 2013] or Wikipedia.…”

“…Ipeirotis and Gabrilovich [2014] study the benefits of displaying scores to workers individually, overall crowd performance, and leaderboards with complete or partial rankings of workers. Leaderboards have been used extensively so far Dontcheva et al 2014;Preist et al 2014].…”

“…To many workers, gaining or improving skills is a motivation per se. This motivation can be enforced by in-person training like in Samasource (http://www.samasource.org/) or Mobileworks (https://www.mobileworks.com/) or through interactive tutorials, such as the ones described by Dontcheva et al [2014]. Also, designing tasks in a way that helps workers learn improves output quality .…”

Toward a Learning Science for Complex Crowdsourcing Tasks

“…Several prior studies explore the training of crowdworkers [22,31,24,7]. Oleson et al proposed the use of gold standards as a form of training on relatively simple microtasks, but their primary focus was on the use of gold standards for quality assurance rather than on quantifying their efficacy in training [22].…”

Suspenseful design: Engaging emotionally with complex applications through compelling narratives

“…Techniques have been devised to create a tutorial by extracting information from a recording of an expert performing a task (Grabler et al, 2009;Chi et al, 2012;Wang et al, 2014), by crowdsourcing the task of tutorial creation (Knabe, 1995;Lafreniere et al, 2013;Dontcheva et al, 2014), or by taking the desired final product and automatically reverse-engineering the process necessary to create it (Li et al, 2013).…”