Adaptive Contract Design for Crowdsourcing Markets: Bandit Algorithms for Repeated Principal-Agent Problems

Ho, Cheng-Maw; Slivkins, Aleksandrs; Vaughan, J.

doi:10.1613/jair.4940

Cited by 68 publications

(67 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In Section 7, we propose a variant of this worker model that is in line with our experimental observations. The results of Ho et al [16] still apply when our worker model is used in place of theirs.…”

Section: Introductionmentioning

confidence: 70%

“…As an example, consider Ho et al [16], a recent theoretical paper on the optimization of PBPs. While that paper posits the standard principal-agent model, all results carry over to our model.…”

Section: Comparison With Principal-agent Modelmentioning

confidence: 99%

“…Ho et al [16] studied the algorithmic problem of adaptively optimizing PBPs in crowdsourcing markets, modeling the problem as a dynamic variant of the standard principalagent model from contract theory. Their model assumed that each worker chooses how much effort to exert in order to maximize his expected utility, which is simply his expected payment minus the cost of his effort.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Incentivizing high quality crowdwork

Slivkins

Suri

et al. 2016

SIGecom Exch.

Self Cite

View full text Add to dashboard Cite

We study the causal effects of financial incentives on the quality of crowdwork. We focus on performance-based payments (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the popular crowdsourcing platform Amazon Mechanical Turk with the goal of understanding when, where, and why PBPs help, identifying properties of the payment, payment structure, and the task itself that make them most effective. We provide examples of tasks for which PBPs do improve quality. For such tasks, the effectiveness of PBPs is not too sensitive to the threshold for quality required to receive the bonus, while the magnitude of the bonus must be large enough to make the reward salient. We also present examples of tasks for which PBPs do not improve quality. Our results suggest that for PBPs to improve quality, the task must be effort-responsive: the task must allow workers to produce higher quality work by exerting more effort. We also give a simple method to determine if a task is effort-responsive a priori. Furthermore, our experiments suggest that all payments on Mechanical Turk are, to some degree, implicitly performance-based in that workers believe their work may be rejected if their performance is sufficiently poor. Finally, we propose a new model of worker behavior that extends the standard principal-agent model from economics to include a worker's subjective beliefs about his likelihood of being paid, and show that the predictions of this model are in line with our experimental findings. This model may be useful as a foundation for theoretical studies of incentives in crowdsourcing markets.

show abstract

Section: Introductionmentioning

confidence: 70%

Section: Comparison With Principal-agent Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Incentivizing high quality crowdwork

Slivkins

Suri

et al. 2016

SIGecom Exch.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The current literature on active crowd-labeling is mainly focused on binary annotation problems (Sheng, Provost, & Ipeirotis, 2008;Donmez & Carbonell, 2008a, 2008bDonmez, Carbonell, & Schneider, 2009;Hsueh, Melville, & Sindhwani, 2009;Welinder & Perona, 2010;Yan, Rosales, Fung, & Dy, 2011;Gao, Liu, Ooi, Wang, & Chen, 2013;Lin, Mausam, & Weld, 2016;Tran-Thanh, Venanzi, Rogers, & Jennings, 2013;Tran-Thanh, Huynh, Rosenfeld, Ramchurn, & Jennings, 2014;Fang, Yin, & Tao, 2014;Raykar & Agrawal, 2014;Mozafari, Sarkar, Franklin, Jordan, & Madden, 2014;Nguyen, Wallace, & Lease, 2015;Zhang, Wen, Tian, Gan, & Wang, 2015;Zhuang & Young, 2015;Zhu, Xu, & Yan, 2015;Ho, Jabbari, & Vaughan, 2013;Ho, Slivkins, & Vaughan, 2016;Khetan & Oh, 2016). We briefly survey the main tenets below.…”

Section: Active Crowd-labeling For Binary Annotation Problemsmentioning

confidence: 99%

“…Their method requires the use of gold standard labels for assessing annotator quality and uses weighted majority voting for inferring the consensus. Ho et al (2016) treat the payment problem for crowdsourcing markets as a multi-armed bandit problem, where each arm represents the contract between a task and an annotator. They propose a method called 'Agnostic Zooming' for selecting the most beneficial contract and study dynamic task pricing.…”

Section: Active Crowd-labeling For Binary Annotation Problemsmentioning

confidence: 99%

Actively Estimating Crowd Annotation Consensus

Kara¹,

Genc²,

Aran³

et al. 2018

jair

View full text Add to dashboard Cite

The rapid growth of storage capacity and processing power has caused machine learning applications to increasingly rely on using immense amounts of labeled data. It has become more important than ever to have fast and inexpensive ways to annotate vast amounts of data. With the emergence of crowdsourcing services, the research direction has gravitated toward putting the wisdom of crowds to better use. Unfortunately, spammers and inattentive annotators pose a threat to the quality and trustworthiness of the consensus. Thus, high quality consensus estimation from crowd annotated data requires a meticulous choice of the candidate annotator and the sample in need of a new annotation. Due to time and budget limitations, it is of utmost importance that this choice is carried out while the annotation collection is in progress. We call this process active crowd-labeling. To this end, we propose an active crowd-labeling approach for actively estimating consensus from continuous-valued crowd annotations. Our method is based on annotator models with unknown parameters, and Bayesian inference is employed to reach a consensus in the form of ordinal, binary, or continuous values. We introduce ranking functions for choosing the candidate annotator and sample pair for requesting an annotation. In addition, we propose a penalizing method for preventing annotator domination, investigate the explore-exploit trade-off for incorporating new annotators into the system, and study the effects of inducing a stopping criterion based on consensus quality. We also introduce the crowd-labeled Head Pose Annotations datasets. Experimental results on the benchmark datasets used in the literature and the Head Pose Annotations datasets suggest that our method provides high-quality consensus by using as few as one fifth of the annotations (∼ 80% cost reduction), thereby providing a budget and time-sensitive solution to the crowd-labeling problem.

show abstract

What Prize Is Right? How to Learn the Optimal Structure for Crowdsourcing Contests

Truong

Stein

Tran-Thanh

et al. 2019

PRICAI 2019: Trends in Artificial Intelligence

View full text Add to dashboard Cite

In crowdsourcing, one effective method for encouraging participants to perform tasks is to run contests where participants compete against each other for rewards. However, there are numerous ways to implement such contests in specific projects. They could vary in their structure (e.g., performance evaluation and the number of prizes) and parameters (e.g., the maximum number of participants and the amount of prize money). Additionally, with a given budget and a time limit, choosing incentives (i.e., contest structures with specific parameter values) that maximise the overall utility is not trivial, as their respective effectiveness in a specific project is usually unknown a priori. Thus, in this paper, we propose a novel algorithm, BOIS (Bayesian-optimisationbased incentive selection), to learn the optimal structure and tune its parameters effectively. In detail, the learning and tuning problems are solved simultaneously by using online learning in combination with Bayesian optimisation. The results of our extensive simulations show that the performance of our algorithm is up to 85% of the optimal and up to 63% better than state-of-the-art benchmarks.

show abstract

Adaptive Contract Design for Crowdsourcing Markets: Bandit Algorithms for Repeated Principal-Agent Problems

Cited by 68 publications

References 49 publications

Incentivizing high quality crowdwork

Incentivizing high quality crowdwork

Actively Estimating Crowd Annotation Consensus

What Prize Is Right? How to Learn the Optimal Structure for Crowdsourcing Contests

Contact Info

Product

Resources

About