Crowdsourcing under Real-Time Constraints

Abstract

In the recent years we are experiencing the rapid growth of crowdsourcing systems, in which "human workers" are enlisted to perform tasks more effectively than computers, and get compensated for the work they provide. The common belief is that the wisdom of the "human crowd" can greatly complement many computer tasks which are assigned to machines. A significant challenge facing these systems is determining the most efficient allocation of tasks to workers to achieve successful completion of the tasks under real-time constraints. This paper presents REACT, a crowdsourcing system that seeks to address this challenge and proposes algorithms that aim to stimulate user participation and handle dynamic task assignment and execution in the crowdsourcing system. The goal is to determine the most appropriate workers to assign incoming tasks, in such a way so that the realtime demands are met and high quality results are returned. We empirically evaluate our approach and show that REACT meets the requested real-time demands, achieves good accuracy, is efficient, and improves the amount of successful tasks that meet their deadlines up to 61% compared to traditional approaches like AMT.

ReLauncher

“…While these approaches help crowdsourcing work easier, they do not address the problem of delays in tasks execution. REACT [2] is a system that dynamically assigns tasks to different workers in order to meet timeliness requirements and to make sure that given quality standards are met. Yet, the approach proposes the matching of workers with tasks based on worker profiles, which only the operators of crowdsourcing platforms have access to and requesters cannot benefit from.…”

Towards Effective Developer Recommendation in Software Crowdsourcing

“…In particular, I. Boutsis and V. Kalogeraki [6] present REACT that schedules tasks for the crowd under time constraints. It collects worker profiles (e.g., real-time computational capabilities) and dynamically assigns tasks to suitable workers.…”

“…The education background includes some [degree, major] tuples. "Edu" is calculated by (6). The work experience includes some [job, duration] tuples.…”

On Task Assignment for Real-Time Reliable Crowdsourcing

“…All experiments in the CRITICAl system were executed on an Intel Core2 Quad Q9300 processor with 4GB of RAM. Since there is no technique, that we know of, that deals with the dual problem of real-time and reliable task assignment in crowdsourcing systems, we compare our approach with: (i)the Traditional approach, where the tasks are assigned to workers in a random manner, and (ii)the REACT Algorithm, developed in our previous work in [33], that aims to determine the optimal selection among workers and tasks to meet real-time criteria only. The REACT algorithm, for a predefined amount of cycles, selects different edges for the matching arbitrarily and accepts the new state if it improves the objective function.…”