A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing

Abstract

Abstract-Assigning heterogeneous tasks to workers is an important challenge of crowdsourcing platforms. Current approaches to task assignment have primarily focused on contentbased approaches, qualifications, or work history. We propose an alternative and complementary approach that focuses on what capabilities workers employ to perform tasks. First, we model various tasks according to the human capabilities required to perform them. Second, we capture the capability traces of the crowd workers performance on existing tasks. Third, we predict performance of workers on new tasks to make task routing decisions, with the help of capability traces. We evaluate the effectiveness of our approach on three different tasks including fact verification, image comparison, and information extraction. The results demonstrate that we can predict worker's performance based on worker capabilities. We also highlight limitations and extensions of the proposed approach.

Investigating the Expertise Indicators of Vulnerability Discovery Professionals

“…image tagging) in order to predict the performance of the crowd worker in tasks of different types (e.g. image recognition) [47]. Sarasua and Matthias however, proposed the use of a crossplatform Curriculum Vitae in their work to record the crowd worker expertise and skills according to the performance of the crowd worker in several crowdsourcing platforms [48].…”

Towards Effective Developer Recommendation in Software Crowdsourcing

“…Most research on crowdsourcing is concentrated on several topics, including how to apply crowdsourcing [2,3], how to control the quality of crowdsourcing [4,5], how to allocate crowdsourcing tasks.…”

“…The original "Bas 0 " is calculated by (5). Suppose that BASE is the maximum value among all the enrolled developers, we divide "Bas 0 " by BASE to get "Bas".…”

A Collaborative Approach for Metadata Management for Internet of Things

“…• Task assignment is concerned with matching between tasks and users of the monitored environment [10]. The matching can be based on characteristics of tasks or the specific requirements of tasks in terms of human capabilities of users [11].…”

“…Whether the pull or push method is used for assignment depends on the data management objective of the task. Therefore data management component must specify the assignment method for tasks, as well as the requirements in terms of human capabilities [11] .…”