Autonomous Crowdsourcing through Human-Machine Collaborative Learning

Abad, Azad; Nabi, Moin; Moschitti, Alessandro

doi:10.1145/3077136.3080666

Cited by 13 publications

(8 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is expected that perceptual learning occurs in microtasks with feedbacks. Abad et al showed that rule-based feedback given to workers who provided incorrect answers is effective for training workers (Abad, Nabi, and Moschitti 2017). Our question is whether this happens even with a simple form of feedback.…”

Section: Related Workmentioning

confidence: 92%

An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks

Kobayashi

Morita

Matsubara

et al. 2018

HCOMP

View full text Add to dashboard Cite

Self-correction for crowdsourced tasks is a two-stage setting that allows a crowd worker to review the task results of other workers; the worker is then given a chance to update his/her results according to the review.Self-correction was proposed as an approach complementary to statistical algorithms in which workers independently perform the same task. It can provide higher-quality results with few additional costs. However, thus far, the effects have only been demonstrated in simulations, and empirical evaluations are needed. In addition, as self-correction gives feedback to workers, an interesting question arises: whether perceptual learning is observed in self-correction tasks. This paper reports our experimental results on self-corrections with a real-world crowdsourcing service.The empirical results show the following: (1) Self-correction is effective for making workers reconsider their judgments. (2) Self-correction is more effective if workers are shown task results produced by higher-quality workers during the second stage. (3) Perceptual learning effect is observed in some cases. Self-correction can give feedback that shows workers how to provide high-quality answers in future tasks.The findings imply that we can construct a positive loop to improve the quality of workers effectively.We also analyze in which cases perceptual learning can be observed with self-correction in crowdsourced microtasks.

show abstract

Section: Related Workmentioning

confidence: 92%

An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks

Kobayashi

Morita

Matsubara

et al. 2018

HCOMP

View full text Add to dashboard Cite

show abstract

“…It is expected that perceptual learning occurs in microtasks with feedbacks. (Abad et al, 2017) showed that rule-based feedback given to workers who provided incorrect answers is useful in training. Our question is whether this happens even with a simple form of feedback.…”

Section: Related Workmentioning

confidence: 99%

Empirical Study on Effects of Self-Correction in Crowdsourced Microtasks

Kobayashi

Morita

Matsubara

et al. 2021

HCj

View full text Add to dashboard Cite

Self-correction for crowdsourced tasks is a two-stage setting that allows a crowd worker to review the task results of other workers; the worker is then given a chance to update their results according to the review.Self-correction was proposed as a complementary approach to statistical algorithms, in which workers independently perform the same task.It can provide higher-quality results with low additional costs. However, thus far, the effects have only been demonstrated in simulations, and empirical evaluations are required.In addition, as self-correction provides feedback to workers, an interesting question arises: whether perceptual learning is observed in self-correction tasks.This paper reports our experimental results on self-corrections with a real-world crowdsourcing service.We found that:(1) Self-correction is effective for making workers reconsider their judgments.(2) Self-correction is effective more if workers are shown the task results of higher-quality workers during the second stage.(3) A perceptual learning effect is observed in some cases. Self-correction can provide feedback that shows workers how to provide high-quality answers in future tasks.(4) A Perceptual learning effect is observed, particularly with workers who moderately change answers in the second stage. This suggests that we can measure the learning potential of workers.These findings imply that requesters/crowdsourcing services can construct a positive loop for improved task results by the self-correction approach.However, (5) no long-term effects of the self-correction task were transferred to other similar tasks in two different settings.

show abstract

“…Research in crowdsourcing has focused on several different issues: aggregating labels from multiple assessors to improve the quality of the gathered assessments, by using unsupervised [Bashir et al, 2013;Hosseini et al, 2012], supervised [Pillai et al, 2013;Raykar and Yu, 2012;Raykar et al, 2010], and hybrid [Harris and Srinivasan, 2013] approaches; behavioural aspects [Kazai et al, 2012b]; proper and careful design of Human Intelligent Tasks (HITs) [Alonso, 2013;Grady and Lease, 2010;Ipeirotis and Gabrilovich, 2014;Kazai et al, 2011], also using gamification to improve quality [Eickhoff et al, 2012] and game theory to increase user engagement [Moshfeghi et al, 2016]; human-machine collaborative methods for training crowdsource workers [Abad, 2017;Abad et al, 2017]; and, routing tasks to proper assessors [Jung and Lease, 2015;Law et al, 2011].…”

Section: Crowdsourcing For Ground-truth Creationmentioning

confidence: 99%

Exploiting User Signals and Stochastic Models to Improve Information Retrieval Systems and Evaluation

Maistro

2019

SIGIR Forum

View full text Add to dashboard Cite

The leitmotiv throughout this thesis is represented by IR evaluation. We discuss different issues related to effectiveness measures and novel solutions that we propose to address these challenges. We start by providing a formal definition of utility-oriented measurement of retrieval effectiveness, based on the representational theory of measurement. The proposed theoretical framework contributes to a better understanding of the problem complexities, separating those due to the inherent problems in comparing systems, from those due to the expected numerical properties of measures. We then propose AWARE, a probabilistic framework for dealing with the noise and inconsistencies introduced when relevance labels are gathered with multiple crowd assessors. By modeling relevance judgements and crowd assessors as sources of uncertainty, we directly combine the performance measures computed on the ground-truth generated by each crowd assessor, instead of adopting a classification technique to merge the labels at pool level. Finally, we investigate evaluation measures able to account for user signals. We propose a new user model based on Markov chains, that allows the user to scan the result list with many degrees of freedom. We exploit this Markovian model in order to inject user models into precision, defining a new family of evaluation measures, and we embed this model as objective function of an LtR algorithm to improve system performances. Table of contents List of figures ix List of tables xi Nomenclature xv Nomenclature xv MP Markov Precision MV Majority Vote nCG normalized Cumulated Gain nDCG normalized Discounted Cumulated Gain nMCG Normalized Markov Cumulated Gain RBP Rank-Biased Precision SERP Search Engine Result Page SMART System for the Mechanical Analysis and Retrieval of Text T REC Text REtrieval Conference With the development of IR systems it became necessary to design a framework to evaluate and compare different retrieval strategies. Indeed, progress and innovation are driven by experiments, but experimentation is useless without an objective evaluation measure that allow researchers to detect the improvements and identify the successful strategies. In Chapter 4 we propose our upstream approach called Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) [Ferrante et al., 2017]. AWARE is defined as an upstream approach because it directly combines the scores of the evaluation measures computed from the relevance labels of each assessor, instead of merging the labels and then computing the measures. The focus is then shifted from the documents and the labels to the evaluation measures. This allows to account for the error introduced by incorrect labels and to develop a framework which estimates performance measures in a way more robust to crowd assessors. Up to now we provide a formal definition of utility-oriented measurement of retrieval effectiveness and we developed an approach to estimate performance measures when there is some noise due to crowd assessors variability. Thus the effective...

show abstract

Autonomous Crowdsourcing through Human-Machine Collaborative Learning

Cited by 13 publications

References 15 publications

An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks

An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks

Empirical Study on Effects of Self-Correction in Crowdsourced Microtasks

Exploiting User Signals and Stochastic Models to Improve Information Retrieval Systems and Evaluation

Contact Info

Product

Resources

About