TurKit: Tools for iterative tasks on mechanical turk

Little, Greg

doi:10.1109/vlhcc.2009.5295247

Cited by 93 publications

(119 citation statements)

References 2 publications

Supporting

Mentioning

113

Contrasting

Unclassified

Order By: Relevance

“…However, aggregating the results of several cheap non-experts, the performance of an expensive professional can be equalled at significantly lower cost. In the same year, Little et al (2009) released TurkIt, a framework for iterative programming of crowdsourcing tasks. In their evaluation, the authors mention relatively low numbers of cheaters.…”

Section: Related Workmentioning

confidence: 99%

Increasing cheat robustness of crowdsourcing tasks

2012

View full text Add to dashboard Cite

Crowdsourcing successfully strives to become a widely used means of collecting large-scale scientific corpora. Many research fields, including Information Retrieval, rely on this novel way of data acquisition. However, it seems to be undermined by a significant share of workers that are primarily interested in producing quick generic answers rather than correct ones in order to optimise their time-efficiency and, in turn, earn more money. Recently, we have seen numerous sophisticated schemes of identifying such workers. Those, however, often require additional resources or introduce artificial limitations to the task. In this work, we take a different approach by investigating means of a priori making crowdsourced tasks more resistant against cheaters.

show abstract

Section: Related Workmentioning

confidence: 99%

Increasing cheat robustness of crowdsourcing tasks

2012

View full text Add to dashboard Cite

show abstract

“…Programming frameworks for human computation such as TurKit [9], Jabberwocky [1], and AutoMan [3] provide highlevel language support for "programming with people", using systems like Amazon Mechanical Turk or social networking platforms like Facebook as their backends. The proposed languages allow some flexibility, for example continuing the computation until a desired confidence level is achieved, but do not offer any facilities for composing complex multi-user workflows and optimising these over time.…”

Section: Related Workmentioning

confidence: 99%

“…Answers, human-based computation platforms such as Amazon Mechanical Turk, and sharing economy applications such as Uber and AirBnB. While specific programming frameworks have been proposed for such applications or to facilitate the use of existing platforms to enable a specific "social computation" [1,3,9,10], the design of generic platforms to orchestrate human-centric collaborative computation has not received much attention in the literature.…”

Section: Introductionmentioning

confidence: 99%

SmartOrch

Rovatsos

Diochnos

Wen

et al. 2017

Proceedings of the Symposium on Applied Computing

View full text Add to dashboard Cite

Web-based collaborative systems, where most computation is performed by human collectives, have distinctly different requirements from traditional workflow orchestration systems, as humans have to be mobilised to perform computations and the system has to adapt to their collective behaviour at runtime. In this paper, we present a social orchestration system called SmartOrch, which has been designed specifically for collective adaptive systems in which human participation is at the core of the overall distributed computation. SmartOrch provides a flexible and customisable workflow composition framework that has multi-level optimisation capabilities. These features allow us to manage the uncertainty that collective adaptive systems need to deal with in a principled way.We demonstrate the benefits of SmartOrch with simulation experiments in a ridesharing domain. Our experiments show that SmartOrch is able to respond flexibly to variation in collective human behaviour, and to adapt to observed behaviour at different levels. This is accomplished by learning how to propose and route human-based tasks, how to allocate computational resources when managing these tasks, and how to adapt the overall interaction model of the platform based on past performance. By proposing novel, solid engineering principles for these kinds of systems, SmartOrch addresses shortcomings of previous work that mostly focused on application-specific, non-adaptive solutions.

show abstract

“…TurKit [22] is a library layered on top of Amazon's Mechanical Turk offering an execution model (crash-and-rerun) which re-offers the same microtasks to the crowd until they are performed satisfactorily. The entire synchronisation, task splitting and aggregation is left entirely to the programmer.…”

Section: Related Workmentioning

confidence: 99%

A Collaboration Model for Community-Based Software Development with Social Machines

Murray-Rust¹,

Šćekić²,

Papapanagiotou³

et al. 2015

EAI Endorsed Transactions on Collaborative Computing

View full text Add to dashboard Cite

Abstract-Today's crowdsourcing systems are predominantly used for processing independent tasks with simplistic coordination. As such, they offer limited support for handling complex, intellectually and organizationally challenging labour types, such as software development. In order to support crowdsourcing of the software development processes, the system needs to enact coordination mechanisms which integrate human creativity with machine support. While workflows can be used to handle highlystructured and predictable labour processes, they are less suitable for software development methodologies where unpredictability is an unavoidable part the process. This is especially true in phases of requirement elicitation and feature development, when both the client and development communities change with time. In this paper we present models and techniques for coordination of human workers in crowdsourced software development environments. The techniques augment the existing Social Compute Unit (SCU) concept-a general framework for management of ad-hoc human worker teams-with versatile coordination protocols expressed in the Lightweight Social Calculus (LSC). This approach allows us to combine coordination and quality constraints with dynamic assessments of software-user's desires, while dynamically choosing appropriate software development coordination models.

show abstract

TurKit: Tools for iterative tasks on mechanical turk

Cited by 93 publications

References 2 publications

Increasing cheat robustness of crowdsourcing tasks

Increasing cheat robustness of crowdsourcing tasks

SmartOrch

A Collaboration Model for Community-Based Software Development with Social Machines

Contact Info

Product

Resources

About