Modelling Cognitive Bias in Crowdsourcing Systems

Saab, Farah; Elhajj, Imad H.; Kayssi, Ayman; Chehab, Ali

doi:10.1016/j.cogsys.2019.04.004

Cited by 24 publications

(16 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Experiment Set D, the highest performance was achieved by the SVM classifier for the BCE-CE input. These results once again indicate that, even though the selfreported confidence values are not particularly helpful when used within the traditional voting methods context (Li and Varshney, 2017;Saab et al, 2019)-as can also be seen by the performance of the CWMV algorithm in this study-incorporating them into an ML classifier can help attain better performance, specifically higher F 1 -scores for highly imbalanced datasets.…”

Section: Performance Of Aggregation Methods On Imbalanced Datasetsmentioning

confidence: 59%

“…Although subjective confidence values can be a valid predictor of accuracy in some cases (Matoulkova, 2017 ; Görzen et al, 2019 ), in many others they may degrade performance owing to cognitive biases that prevent a realistic assessment of one's abilities (Saab et al, 2019 ). Another natural approach is to weigh responses based on some form of worker reliability.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning

Yasmin

Hassan²,

Grassel³

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

This work investigates how different forms of input elicitation obtained from crowdsourcing can be utilized to improve the quality of inferred labels for image classification tasks, where an image must be labeled as either positive or negative depending on the presence/absence of a specified object. Five types of input elicitation methods are tested: binary classification (positive or negative); the (x, y)-coordinate of the position participants believe a target object is located; level of confidence in binary response (on a scale from 0 to 100%); what participants believe the majority of the other participants' binary classification is; and participant's perceived difficulty level of the task (on a discrete scale). We design two crowdsourcing studies to test the performance of a variety of input elicitation methods and utilize data from over 300 participants. Various existing voting and machine learning (ML) methods are applied to make the best use of these inputs. In an effort to assess their performance on classification tasks of varying difficulty, a systematic synthetic image generation process is developed. Each generated image combines items from the MPEG-7 Core Experiment CE-Shape-1 Test Set into a single image using multiple parameters (e.g., density, transparency, etc.) and may or may not contain a target object. The difficulty of these images is validated by the performance of an automated image classification method. Experiment results suggest that more accurate results can be achieved with smaller training datasets when both the crowdsourced binary classification labels and the average of the self-reported confidence values in these labels are used as features for the ML classifiers. Moreover, when a relatively larger properly annotated dataset is available, in some cases augmenting these ML algorithms with the results (i.e., probability of outcome) from an automated classifier can achieve even higher performance than what can be obtained by using any one of the individual classifiers. Lastly, supplementary analysis of the collected data demonstrates that other performance metrics of interest, namely reduced false-negative rates, can be prioritized through special modifications of the proposed aggregation methods.

show abstract

Section: Performance Of Aggregation Methods On Imbalanced Datasetsmentioning

confidence: 59%

Section: Literature Reviewmentioning

confidence: 99%

Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning

Yasmin

Hassan²,

Grassel³

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…self-efficacy, effort, and performance in crowdsourcing teams 1 1 0 [12] Cooperation or competition-When do people contribute more? A field experiment on gamification of crowdsourcing 1 1 0 [11] Crowdsourcing: A taxonomy and systematic mapping study 1 1 0 [55] Crowdsourcing contests 1 1 0 [21] Crowdsourcing not all sourced by the crowd: An observation on the behavior of Wikipedia participants 1 1 0 [56] Efficient crowdsourcing of unknown experts using bounded multiarmed bandits 1 1 1 [57] Hybrid crowd-based decision support in business processes 1 1 1 [58] Improving accuracy and lowering cost in crowdsourcing through an unsupervised expertise estimation approach 1 0 1 [59] Incentivizing social media users for mobile crowdsourcing 1 1 1 [60] Information technology (IT)-enabled crowdsourcing: A conceptual framework 1 1 1 [61] Inspiring crowdsourcing communities to create novel solutions: Competition design and the mediating role of trust 1 1 0 [62] Mobile crowd sensing-Taxonomy, applications, challenges, and solutions 0 1 1 [63] Modeling cognitive bias in crowdsourcing systems 1 1 0 [64] Open or proprietary? Choosing the right crowdsourcing platform for innovation 1 1 0 [23] Privacy-preserving QoI-aware participant coordination for mobile crowdsourcing 0 1 1 [2] Real-time crowdsourcing with payment of idle workers in the retainer model 1 1 0 [65] SenseChain: A blockchain-based crowdsensing framework for multiple requesters and multiple workers 0 1 1 [14] e wisdom of crowds: e potential of online communities as a tool for data analysis 1 1 0 [30] Toward collaborative software engineering leveraging the crowd 1 1 0 [8] Trait motivations of crowdsourcing and task choice: A distal-proximal perspective 1 1 0 [9] Trust-based privacy-aware participant selection in social participatory sensing 0 1 1 8 Scientific Programming in gaining the rewards that are associated with tasks and thus do not work sincerely, and this is a negative factor of the crowd [29].…”

Section: Significant Features Of Crowd In Crowdsourcing (Rq1)mentioning

confidence: 99%

“…Trainees may accomplish tasks [3,41] in the crowdsourcing activity. e crowd may be autonomous [34], fast [2] unique [69], appropriate [3], right [28,79], reliable/efficient [2,25,72,73,80], loyal [70], truthful [36], trustworthy [29,42,44,50,63] who complete assigned tasks sincerely. Workers are coordinative, adaptable [16] (as they can change themselves with the change in the work environment), "Energetic" having energy [10,17], capable [1,42,60] of performing tasks, and are "creative" as they creatively [67,68,78] perform different types of work.…”

Section: Significant Features Of Crowd In Crowdsourcing (Rq1)mentioning

confidence: 99%

“…Workers are coordinative, adaptable [16] (as they can change themselves with the change in the work environment), "Energetic" having energy [10,17], capable [1,42,60] of performing tasks, and are "creative" as they creatively [67,68,78] perform different types of work. ey are competent workers [16,39,63] who utilize common sense [68] in various tasks. Crowds are smart [25,27], educated/ qualified [1,13,42], professionals [71,75], having skills (skillful workers) [2, 8, 10, 12, 14, 17, 26, 28, 37, 48, 51, 54, 56, 57, 59-61, 67, 76-78], expertise [58,59,81,82], experience [14,17,19,24,43,46,47,53,76] who possess knowledge [2, 8, 11, 13-15, 17, 19, 22, 30, 35, 38, 49, 55, 60, 61, 68, 70, 71, 75, 76] for getting a task done [35].…”

Section: Significant Features Of Crowd In Crowdsourcing (Rq1)mentioning

confidence: 99%

See 1 more Smart Citation

Selection of Crowd in Crowdsourcing for Smart Intelligent Applications: A Systematic Mapping Study

Shang

Ali

et al. 2021

Scientific Programming

View full text Add to dashboard Cite

Crowdsourcing is a task-solving model in which human crowd is hired to solve a particular task. During the crowdsourcing process, the crowd selection is performed in order to select appropriate crowd workers for a specific task; without appropriate selection of crowd workers, the process of crowdsourcing is aimless. The main goal of this paper was to identify the features of crowd in crowdsourcing activity, reasons behind crowd participation in the activity of crowdsourcing, and the existing techniques that were utilized for crowd selection in crowdsourcing. Search strings with corresponding keywords were used to capture relevant studies related to crowdsourcing, and crowd selection was classified under conference papers, journal articles, proceedings, and book chapters. 81 relevant studies are selected from 7 digital data repositories using a search strategy. In crowdsourcing practices, crowd selection was considerably addressed. Nonetheless, it has been noticed that the selection is based only on a single crowd worker attribute such as confidence, past success, efficiency, and experience. For the efficiency and effectiveness of the crowdsourcing operation, crowd selection on multicriteria features is essential.

show abstract