MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture

Kulkarni, Anand; Gutheim, Philipp; Narula, Prayag; Rolnitzky, David; Parikh, Tapan S.; Hartmann, Björn

doi:10.1109/mic.2012.72

Cited by 73 publications

(47 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Over 10 million people use Duolingo to learn a language by translating small snippets of text; aggregating these translations produces translations of websites and other documents [2]. Building on the broad use of Amazon Mechanical Turk, firms such as MobileWorks and CrowdFlower pro 1 2 vide a managed environment to enable clients to complete often urgent work with a crowd [14].…”

Section: Introductionmentioning

confidence: 99%

Borrowing from the Crowd: A Study of Recombination in Software Design Competitions

LaToza

Chen

Jiang

et al. 2015

2015 IEEE/ACM 37th IEEE International Conference on Software Engineering

View full text Add to dashboard Cite

Abstract-Crowdsourcing has had extraordinary success in solving a diverse set of problems, ranging from digitization of libraries and translation of the Internet, to scientific challenges such as classifying elements in the galaxy or determining the 3D shape of an enzyme. By leveraging the power of the masses, it is feasible to complete tasks in mere days and sometimes even hours, and to take on tasks that were previously impossible because of their sheer scale. Underlying the success of crowdsourcing is a common theme -the microtask. By breaking down the overall task at hand into microtasks providing short, self-contained pieces of work, work can be performed independently, quickly, and in parallel -enabling numerous and often untrained participants to chip in. This paper puts forth a research agenda, examining the question of whether the same kinds of successes that microtask crowdsourcing is having in revolutionizing other domains can be brought to software development. That is, we ask whether it is possible to push well beyond the open source paradigm, which still relies on traditional, coarse-grained tasks, to a model in which programming proceeds through microtasks performed by vast numbers of crowd developers.

show abstract

Section: Introductionmentioning

confidence: 99%

Borrowing from the Crowd: A Study of Recombination in Software Design Competitions

LaToza

Chen

Jiang

et al. 2015

2015 IEEE/ACM 37th IEEE International Conference on Software Engineering

View full text Add to dashboard Cite

show abstract

“…Tokarchuk et al [2012] state that clarity positively correlates with performance, a result that is empirically confirmed by Georgescu et al [2012], while Kulkarni et al [2012b] specifically study the problem of incomplete descriptions. Needless to say, if a worker does not immediately understand a task, he will not be willing to work on it.…”

Section: Task Description Qualitymentioning

confidence: 69%

“…If both the skills required by a task and the ones possessed by workers are defined, tasks can be automatically assigned. For example, Mobileworks uses priority queues to assign tasks [Kulkarni et al 2012b]. assign stronger workers to harder tasks, while Roy et al [2015] specifically focus on knowledge-intensive tasks and show the benefits of maintaining a pre-computed index for efficient worker-totask assignment.…”

Section: Reject Workersmentioning

confidence: 99%

“…Hu et al [2012], for instance, place widgets with integrated tasks on third-party services, such as the International Children's Digital Library. Kulkarni et al [2012b] ask workers to recruit others for a given task, others incentivize this kind of recruitment [Nath et al 2012]. Also posting task information on worker community sites, such as the "HITsWorthTurkingFor" reddit page, can increase the task's workforce.…”

Section: Promote Tasksmentioning

confidence: 99%

“…Clarity Hossfeld et al [2014], Tokarchuk et al [2012], Georgescu et al [2012], Kulkarni et al [2012b] Complexity Hu et al [2012, Rogstadius et al [2011], User interf. Usability , Noronha et al [2011], Retelny et al [2014] …”

Section: Taskmentioning

confidence: 99%

See 2 more Smart Citations

Quality Control in Crowdsourcing

et al. 2018

View full text Add to dashboard Cite

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar -all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.

show abstract

Crowdsourcing and Human Computation, Introduction

Lease¹,

Alonso²

2014

Encyclopedia of Social Network Analysis and Mining

View full text Add to dashboard Cite

MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture

Cited by 73 publications

References 13 publications

Borrowing from the Crowd: A Study of Recombination in Software Design Competitions

Borrowing from the Crowd: A Study of Recombination in Software Design Competitions

Quality Control in Crowdsourcing

Crowdsourcing and Human Computation, Introduction

Contact Info

Product

Resources

About