A Survey on Task Assignment in Crowdsourcing

Hettiachchi, Danula; Kostakos, Vassilis; Gonçalves, Jorge

doi:10.1145/3494522

Cited by 39 publications

(11 citation statements)

References 155 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The common solution is to select the N workers with the best performance, while the worker performance can be evaluated from historical worker data, or gold standard questions (questions with known answers) (Hettiachchi et al , 2022), or estimated by probabilistic inference algorithms such as expectation maximization (Whitehill et al , 2009). However, in addition to avg_p and best , the performance of aggregation results is significantly affected by other factors, for example, dist (a representation of the concept of group diversity).…”

Section: Methodsmentioning

confidence: 99%

“…Li and Liu (2015) pointed out that worker performance and variance are important factors for classification task. Hettiachchi et al (2022) reviewed the existing worker selection models. However, the existing work does not research how to select the optimal working group with a given number of workers to optimize the performance of the aggregated results.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Performance prediction of multivariable linear regression based on the optimal influencing factors for ranking aggregation in crowdsourcing task

Xing

Zhan

2023

DTA

View full text Add to dashboard Cite

PurposeFor ranking aggregation in crowdsourcing task, the key issue is how to select the optimal working group with a given number of workers to optimize the performance of their aggregation. Performance prediction for ranking aggregation can solve this issue effectively. However, the performance prediction effect for ranking aggregation varies greatly due to the different influencing factors selected. Although questions on why and how data fusion methods perform well have been thoroughly discussed in the past, there is a lack of insight about how to select influencing factors to predict the performance and how much can be improved of.Design/methodology/approachIn this paper, performance prediction of multivariable linear regression based on the optimal influencing factors for ranking aggregation in crowdsourcing task is studied. An influencing factor optimization selection method based on stepwise regression (IFOS-SR) is proposed to screen the optimal influencing factors. A working group selection model based on the optimal influencing factors is built to select the optimal working group with a given number of workers.FindingsThe proposed approach can identify the optimal influencing factors of ranking aggregation, predict the aggregation performance more accurately than the state-of-the-art methods and select the optimal working group with a given number of workers.Originality/valueTo find out under which condition data fusion method may lead to performance improvement for ranking aggregation in crowdsourcing task, the optimal influencing factors are identified by the IFOS-SR method. This paper presents an analysis of the behavior of the linear combination method and the CombSUM method based on the optimal influencing factors, and optimizes the task assignment with a given number of workers by the optimal working group selection method.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Performance prediction of multivariable linear regression based on the optimal influencing factors for ranking aggregation in crowdsourcing task

Xing

Zhan

2023

DTA

View full text Add to dashboard Cite

show abstract

“…Again, we will focus on summarizing those that are most closely related to ours. Some papers, such as [Hettiachchi et al(2022)], [Zhang et al(2017)] study how to assign different crowd workers to multiple tasks in order to maximize the expected accuracy of labels.…”

Section: Crowdsourcingmentioning

confidence: 99%

Private Blotto: Viewpoint Competition with Polarized Agents

Donahue¹,

Kleinberg²

2023

Preprint

View full text Add to dashboard Cite

Colonel Blotto games are one of the oldest settings in game theory, originally proposed over a century ago in [Borel(1921)]. However, they were originally designed to model two centrally-controlled armies competing over zero-sum "fronts", a specific scenario with limited modern-day application. In this work, we propose and study Private Blotto games, a variant connected to crowdsourcing and social media. One key difference in Private Blotto is that individual agents act independently, without being coordinated by a central "Colonel". This model naturally arises from scenarios such as activist groups competing over multiple issues, partisan fund-raisers competing over elections in multiple states, or politically-biased social media users labeling news articles as misinformation. In this work, we completely characterize the Nash Stability of the Private Blotto game. Specifically, we show that the outcome function has a critical impact on the outcome of the game: we study whether a front is won by majority rule (median outcome) or a smoother outcome taking into account all agents (mean outcome). We study how this impacts the amount of "misallocated effort", or agents whose choices doesn't influence the final outcome. In general, mean outcome ensures that, if a stable arrangement exists, agents are close to evenly spaced across fronts, minimizing misallocated effort. However, mean outcome functions also have chaotic patterns as to when stable arrangements do and do not exist. For median outcome, we exactly characterize when a stable arrangement exists, but show that this outcome function frequently results in extremely unbalanced allocation of agents across fronts.

show abstract

“…2015); task assignment based on individual difference measures (Hettiachchi et al. 2020). Building on earlier work by Kahneman and colleagues, a number of studies have also addressed biases in crowdsourced judgments, for example, Eickhoff (2018) and Draws et al.…”

Section: How Can the Ai/ml Community Deal With Noise?mentioning

confidence: 99%

Groups of experts often differ in their decisions: What are the implications for AI and machine learning? A commentary onNoise: A Flaw in Human Judgment, byKahneman, Sibony, and Sunstein (2021)

Sleeman,

Gilhooly

2023

AI Magazine

View full text Add to dashboard Cite

Machine Learning systems rely heavily on annotated instances. Such annotations are frequently done by human experts, or by tools developed by experts, and so the central message of this book, Noise: A Flaw in Human Judgment (Kahneman, Sibony, and Sunstein 2021) is of considerable importance to AI/Machine Learning community. The core message is that if a number of experts are asked to annotate tasks that involve judgments, these responses will frequently differ. This observation poses a problem for how analysts choose a particular annotated dataset (from the group), or process the set of responses to give a “balanced” response, or whether to reject all the annotated datasets. A further important aspect of this book is the case studies which demonstrate that differences in judgments between fellow experts have been reported in a significant number of disciplines including, business, the law, government, and medicine. Kahneman, Sibony and Sunstein (2021), referred to as KSS subsequently, discuss how Expert Biases can be reduced, but the main focus of this book is a discussion of Noise, that is, differences that often occur between fellow experts, and how Noise can often be reduced. To address the last point KSS have formulated a set of six decision hygiene principles which include the recommendation that complex tasks should be subdivided, and then each subtask should be solved separately. A further principle is that each task should be solved by individual experts before the various judgments are discussed with fellow experts. Effectively, the book being reviewed covers three main topics: First, it reports several motivating studies that show how judgments of fellow experts varied significantly in the pricing of insurance premiums, and in setting the lengths of custodial sentences. These motivating studies very effectively illustrate the central concepts of Judgment, Noise, and Bias; that section also provides definitions of these core concepts and discusses how Noise is often amplified in group meetings. Secondly, the authors provide detailed discussion of further studies, in a variety of domains, which report the levels of disagreement between experts. Thirdly, KSS discusses how to reduce the levels of Noise between experts, as noted above, the authors refer to these as Principles of Noise Hygiene. These three parts are interwoven in a complex way throughout the book; in our view, the best overview of the book is given in the section Review and Conclusions: Taking Noise Seriously (KSS, p. 361).

show abstract

A Survey on Task Assignment in Crowdsourcing

Cited by 39 publications

References 155 publications

Performance prediction of multivariable linear regression based on the optimal influencing factors for ranking aggregation in crowdsourcing task

Performance prediction of multivariable linear regression based on the optimal influencing factors for ranking aggregation in crowdsourcing task

Private Blotto: Viewpoint Competition with Polarized Agents

Groups of experts often differ in their decisions: What are the implications for AI and machine learning? A commentary onNoise: A Flaw in Human Judgment, byKahneman, Sibony, and Sunstein (2021)

Contact Info

Product

Resources

About