Improving Reactions to Rejection in Crowdsourcing Through Self-Reflection

Abstract

In popular crowdsourcing marketplaces like Amazon Mechanical Turk, crowd workers complete tasks posted by requesters in return for monetary rewards. Task requesters are solely responsible for deciding whether to accept or reject submitted work. Rejecting work can directly affect the monetary reward of corresponding workers, and indirectly influence worker qualifications and their future work opportunities in the marketplace. Unexpected or unwarranted rejections therefore result in negative emotions and reactions among workers. Despite the high prevalence of rejections in crowdsourcing marketplaces, little research has explored ways to mitigate the negative emotional repercussions of rejections on crowd workers. Addressing this important research gap, we investigate whether introducing self-reflection at different stages after task execution can alleviate the emotional toll of rejection decisions on crowd workers. Our work is inspired by prior studies in psychology that have shown that self-reflection on negative personal experiences can positively affect one's emotion. To this end, we carried out an experimental study investigating the impact of explicit self-reflection on the emotions of rejected crowd workers. Results show that allowing workers to self-reflect on their delivered work, especially before receiving a rejection, has a significantly positive impact on their self-reported emotions in terms of valence and dominance. Our findings reveal that introducing a self-reflection stage before workers receive acceptance or rejection decisions on submitted work, can help in positively influencing the emotions of a worker. These findings have important design implications towards fostering a healthier requester-worker relationship and contributing towards the sustainability of the crowdsourcing marketplace.

A Checklist to Combat Cognitive Biases in Crowdsourcing

“…Have crowd workers who commit to my task, been properly informed about the consequences of their participation? The task selection process is often fairly arbitrary, which means that workers may not realize potential negative effects of committing to a task that they don't have expertise on (Edixhoven et al 2021). For example, crowd workers may commit to doing "paella pan"relevance judgments for products on a whim without considering the potential reputation loss and bad annotation quality that could follow if they do not perform well.…”

What You Show is What You Get! Gestures for Microtask Crowdsourcing

“…Mental wellbeing has been shown to be closely related to the physical wellbeing of people in a variety of contexts [22]. Considering that crowd workers deal with a variety of challenges that impede their mental wellbeing, ranging from unfair treatment to cognitive exhaustion [7,19], or exposure to explicit or disturbing content [15,28], it is important to consider the impact of repetitive work in monotonous task batches on their physical health. At this juncture we argue in the potential of considering the integration of a novel input modality that can serve the dual purpose of facilitating better physical health among workers while accommodating the primary need for input elicitation.…”