A Checklist to Combat Cognitive Biases in Crowdsourcing

A wealth of evidence across several domains indicates that goal setting improves performance and learning by enabling individuals to commit their thoughts and actions to goal achievement. Recently, researchers have begun studying the effects of goal setting in paid crowdsourcing to improve the quality and quantity of contributions, increase learning gains, and hold participants accountable for contributing more effectively. However, there is a lack of research addressing crowd workers' goal-setting practices, how they are currently pursuing them, and the challenges that they face. This information is essential for researchers and developers to create tools that assist crowd workers in pursuing their goals more effectively, thereby improving the quality of their contributions. This paper addresses these gaps by conducting mixed-method research in which we surveyed 205 workers from two crowdsourcing platforms -- Amazon Mechanical Turk (MTurk) and Prolific -- about their goal-setting practices. Through a 14-item survey, we asked workers regarding the types of goals they create, their goal achievement strategies, potential barriers that impede goal attainment, and their use of software tools for effective goal management. We discovered that (a) workers actively create intrinsic and extrinsic goals; (b) use a combination of tools for goal management; (c) medical issues and a busy lifestyle are some obstacles to their goal achievement; and (d) we gathered novel features for future goal management tools. Our findings shed light on the broader implications of developing goal management tools to improve workers' well-being.

Section: Methodsmentioning

confidence: 99%

Goal-Setting Behavior of Workers on Crowdsourcing Platforms: An Exploratory Study on MTurk and Prolific

Abbas

Gadiraju

2022

“…Crowd workers can perform as well as domain experts in certain tasks (Snow et al 2008), especially when the composition of the workers' pool is carefully curated in a task-dependant manner (Allen et al 2021). However, researchers have brought attention to the risk of annotators' cognitive bias (Eickhoff 2018;Draws et al 2021), stereotyping encoded in annotators' assessments (Otterbacher 2015), and uneven representations of demographic characteristics among annotators (Barbosa and Chen 2019). A number of factors, including task and instructions clarity (Wu and Quinn 2017), task design (Kazai et al 2011), incentives (Shah and Zhou 2015), and quality control mechanisms (Ipeirotis, Provost, and Wang 2010;McDonnell et al 2016), have been demonstrated to affect the quality of the annotations (Draws et al 2021).…”

Section: Related Workmentioning

confidence: 99%

“…However, researchers have brought attention to the risk of annotators' cognitive bias (Eickhoff 2018;Draws et al 2021), stereotyping encoded in annotators' assessments (Otterbacher 2015), and uneven representations of demographic characteristics among annotators (Barbosa and Chen 2019). A number of factors, including task and instructions clarity (Wu and Quinn 2017), task design (Kazai et al 2011), incentives (Shah and Zhou 2015), and quality control mechanisms (Ipeirotis, Provost, and Wang 2010;McDonnell et al 2016), have been demonstrated to affect the quality of the annotations (Draws et al 2021). Furthermore, even when labels are collected from domain experts, this does not mean they are free of bias.…”

Section: Related Workmentioning

confidence: 99%

When More Data Lead Us Astray: Active Data Acquisition in the Presence of Label Bias

De‐Arteaga

Saar‐Tsechansky

2022

An increased awareness concerning risks of algorithmic bias has driven a surge of efforts around bias mitigation strategies. A vast majority of the proposed approaches fall under one of two categories: (1) imposing algorithmic fairness constraints on predictive models, and (2) collecting additional training samples. Most recently and at the intersection of these two categories, methods that propose active learning under fairness constraints have been developed. However, proposed bias mitigation strategies typically overlook the bias presented in the observed labels. In this work, we study fairness considerations of active data collection strategies in the presence of label bias. We first present an overview of different types of label bias in the context of supervised learning systems. We then empirically show that, when overlooking label bias, collecting more data can aggravate bias, and imposing fairness constraints that rely on the observed labels in the data collection process may not address the problem. Our results illustrate the unintended consequences of deploying a model that attempts to mitigate a single type of bias while neglecting others, emphasizing the importance of explicitly differentiating between the types of bias that fairness-aware algorithms aim to address, and highlighting the risks of neglecting label bias during data collection.

“…Their findings reveal that workers with strong opinions tend to produce biased annotations and such bias should be mitigated to improve the quality of the data collected. In (Draws et al 2021), the authors propose a 12-item checklist adapted from business psychology to combat cognitive biases in crowdsourcing. We utilize this checklist to point out the potential biases in the data collected in our study out of the 12 items.…”

Section: Potential Biasesmentioning

confidence: 99%

SignUpCrowd: Using Sign-Language as an Input Modality for Microtask Crowdsourcing

Singh

Wehkamp²,

Gadiraju

2022

Different input modalities have been proposed and employed in technological landscapes like microtask crowdsourcing. However, sign language remains an input modality that has received little attention. Despite the fact that thousands of people around the world primarily use sign language, very little has been done to include them in such technological landscapes. We aim to address this gap and take a step towards the inclusion of deaf and mute people in microtask crowdsourcing. We first identify various microtasks which can be adapted to use sign language as input, while elucidating the challenges it introduces. We built a system called ‘SignUpCrowd’ that can be used to support sign language input for microtask crowdsourcing. We carried out a between-subjects study (N=240) to understand the effectiveness of sign language as an input modality for microtask crowdsourcing in comparison to prevalent textual and click input modalities. We explored this through the lens of visual question answering and sentiment analysis tasks by recruiting workers from the Prolific crowdsourcing platform. Our results indicate that sign language as an input modality in microtask crowdsourcing is comparable to the prevalent standards of using text and click input. Although people with no knowledge of sign language found it difficult to use, this input modality has the potential to broaden participation in crowd work. We highlight evidence suggesting the scope for sign language as a viable input type for microtask crowdsourcing. Our findings pave the way for further research to introduce sign language in real-world applications and create an inclusive technological landscape that more people can benefit from.