Task Assignment Optimization in Collaborative Crowdsourcing

Abstract

A number of emerging applications, such as, collaborative document editing, sentence translation, and citizen journalism require workers with complementary skills and expertise to form groups and collaborate on complex tasks. While existing research has investigated task assignment for knowledge intensive crowdsourcing, they often ignore the aspect of collaboration among workers, that is central to the success of such tasks. Research in behavioral psychology has indicated that large groups hinder successful collaboration. Taking that into consideration, our work is one of the first to investigate and formalize the notion of collaboration among workers and present theoretical analyses to understand the hardness of optimizing task assignment. We propose efficient approximation algorithms with provable theoretical guarantees and demonstrate the superiority of our algorithms through a comprehensive set of experiments using real-world and synthetic datasets. Finally, we conduct a real world collaborative sentence translation application using Amazon Mechanical Turk that we hope provides a template for evaluating collaborative crowdsourcing tasks in micro-task based crowdsourcing platforms.

Collaborative crowdsourcing with crowd4U

“…Its key innovations are the reliance of a task-specific assignment algorithm to 1 crowd4u.org find the best group of workers to complete a task, and the implementation of sequential, simultaneous and hybrid worker collaboration, to ensure effective result coordination. Our task assignment algorithm is scalable [9] and can handle multiple tasks and users. We describe how task assignment and result coordination are achieved.…”

“…The assignment algorithm combines different human factor such as workers' skills, worker-to-worker affinity and upper critical mass [9], to find the right team of workers to complete a task. The worker manager supplies the CyLog processor and the task assignment manager with worker human factors (e.g., languages, countries, and applicationspecific human factors) and the relationship among workers encoded in the worker affinity matrix, which maintains the information on how a pair of workers is expected to work well.…”

“…Requesters can specify collaborative tasks in a declarative manner. Tasks are then assigned to groups of workers taking human factors [11,9] and worker-to-worker affinity into account. Crowd4U can use any task decomposition algorithm and handles various worker collaboration schemes that ensure effective result coordination.…”

“…Worker collaboration takes one of three forms: sequential as in text translation [9] where workers improve each others' contributions; simultaneous, as in citizen journalism, where workers generate content in parallel; or hybrid, as in surveillance tasks, where some workers contribute to fact collection in a sequence, correcting each others' observations, and others provide testimonials separately and simultaneously. Figure 1 summarizes the process of deploying complex tasks.…”

“…Its key innovations are the reliance of a task-specific assignment algorithm to find the best group of workers to complete a task, and the implementation of sequential, simultaneous and hybrid worker collaboration schemes, to ensure effective result coordination. Depending on the nature of the task, the assignment algorithm combines different human factors such as workers' skills, worker-to-worker affinity and upper critical mass [9]. Skills are used to filter out unqualified workers.…”