Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks

This paper reports outcomes of a systematic scoping review of methodological approaches and analytical lenses used in empirical research on crowdwork. Over the past decade a growing corpus of publications spanning Social Sciences and Computer Science/HCI have empirically examined the nature of work practices and tasks within crowdwork; surfaced key individual and environmental factors underpinning workers’ decisions to engage in this form of work; developed and implemented tools to improve and extend various aspects of crowdwork, such as the design and allocation of tasks and incentives or workflows within the platforms; and contributed new techniques and know-how on data collection within crowdwork, for example, how to conduct large-scale surveys and experiments in behavioural psychology, economics or education drawing on crowdworker samples. Our initial reading of the crowdwork literature suggested that research had relied on a limited set of relatively narrow methodological approaches, mostly online experiments, surveys and interviews. Importantly, crowdwork research has tended to examine workers’ experiences as snapshots in time rather than studying these longitudinally or contextualising them historically, environmentally and developmentally. This piece-meal approach has given the research community initial descriptions and interpretations of crowdwork practices and provided an important starting point in a nascent field of study. However, the depth of research in the various areas, and the missing pieces, have yet to be systematically scoped out. Therefore, this paper systematically reviews the analytical-methodological approaches used in crowdwork research identifying gaps in these approaches. We argue that to take crowdwork research to the next level it is essential to examine crowdwork practices within the context of both individual and historical-environmental factors impacting it. To this end, methodological approaches that bridge sociological, psychological, individual, collective, online, offline, and temporal processes and practices of crowdwork are needed. The paper proposes the Life Course perspective as an interdisciplinary framework that can help address these gaps and advance research on crowdwork. The paper concludes by proposing a set of Life Course-inspired research questions to guide future studies of crowdwork.

show abstract

“…• Self-assessment theory (Gadiraju et al, 2016) • Equity Theory (d'Eon et al, 2019) • No analytical lens (n=12)…”

Section: Experiments (20% 14/70)mentioning

confidence: 99%

The Life Course: An interdisciplinary framework for broadening the scope of research on crowdwork

Margaryan

Hofmeister

2021

HCj

View full text Add to dashboard Cite

show abstract

“…It is known that assessments by workers are biased. (Gadiraju et al, 2017) showed that crowd workers often lack awareness about their actual level of competence. Incorporating such bias into a self-correction framework would be a challenge for future work.…”

Section: Related Workmentioning

confidence: 99%

Empirical Study on Effects of Self-Correction in Crowdsourced Microtasks

Kobayashi

Morita

Matsubara

et al. 2021

HCj

View full text Add to dashboard Cite

Self-correction for crowdsourced tasks is a two-stage setting that allows a crowd worker to review the task results of other workers; the worker is then given a chance to update their results according to the review.Self-correction was proposed as a complementary approach to statistical algorithms, in which workers independently perform the same task.It can provide higher-quality results with low additional costs. However, thus far, the effects have only been demonstrated in simulations, and empirical evaluations are required.In addition, as self-correction provides feedback to workers, an interesting question arises: whether perceptual learning is observed in self-correction tasks.This paper reports our experimental results on self-corrections with a real-world crowdsourcing service.We found that:(1) Self-correction is effective for making workers reconsider their judgments.(2) Self-correction is effective more if workers are shown the task results of higher-quality workers during the second stage.(3) A perceptual learning effect is observed in some cases. Self-correction can provide feedback that shows workers how to provide high-quality answers in future tasks.(4) A Perceptual learning effect is observed, particularly with workers who moderately change answers in the second stage. This suggests that we can measure the learning potential of workers.These findings imply that requesters/crowdsourcing services can construct a positive loop for improved task results by the self-correction approach.However, (5) no long-term effects of the self-correction task were transferred to other similar tasks in two different settings.

show abstract

“…Workers spend more than 25% of their time searching for tasks that they want to complete [KDM*14]. Thus, in the future, it would be helpful for crowdsourcing platforms to either order the tasks based on their cognitive demands [CIT16] or automatically fetch and suggest tasks to the worker [LRMB15, CIT16, GCB16] based on a self‐assessment report filled in by the worker [GFK*17]. If such a self‐assessment report also includes the results of the ability tests (e.g., colorblindness, visual literacy) that the worker has completed for previous tasks, and these reports are accessible to all experimenters, then experiments could be shorter, more focused on the assessed task, and less cumbersome for the worker.…”

Section: Opportunities and Challengesmentioning

confidence: 99%

Information Visualization Evaluation Using Crowdsourcing

Borgo

Micallef

Bach

et al. 2018

Computer Graphics Forum

View full text Add to dashboard Cite

Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research.

show abstract

Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks

Cited by 48 publications

References 35 publications

The Life Course: An interdisciplinary framework for broadening the scope of research on crowdwork

The Life Course: An interdisciplinary framework for broadening the scope of research on crowdwork

Empirical Study on Effects of Self-Correction in Crowdsourced Microtasks

Information Visualization Evaluation Using Crowdsourcing

Contact Info

Product

Resources

About