Shepherding the crowd yields better work

Abstract

Micro-task platforms provide massively parallel, ondemand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, taskspecific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.

Quality Control in Crowdsourcing Systems: Issues and Directions

“…7,12 Knowing about evaluation and compensation policies helps workers align their work based on these criteria and produce contributions with higher quality. 12 We broadly categorize incentives into two types: intrinsic incentives, such as personal enthusiasm or altruism, and extrinsic incentives, such as monetar y reward. Intrinsic incentives in conjunction with extrinsic ones can motivate honest users to participate in the task.…”

Crowd-Enhanced Cloud Services: Issues and Directions

“…Moreover, workers may increase their experience while contributing to a task by receiving real-time feedback from other workers or requester. The feedback received in real-time, and before final submission of the worker's contribution, can assist the worker to pre-assess the provided contribution and enhance the contribution to satisfy the task requirements more fully [41]. Real-time workflow control and feedback can directly impact the outcome quality, the execution time and also the cost of the task, so they should be taken into account when studying crowdsourcing processes.…”

Combining crowdsourcing and learning to improve engagement and performance

“…In our work, we build on previous techniques used in crowdsourcing creative work [4,8,12,13,14,16,20] and sug gest that a learning framework may further enhance creative crowd work. When the task to be crowdsourced is presented in the context of learning new skills, workers should be less likely to cheat or produce poor quality results; doing so not only harms the requester, it also subverts the learning objec tives of the worker.…”