Modus Operandi of Crowd Workers

Abstract

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that of work environments; defined as the hardware and software affordances at the disposal of crowd workers which are used to complete microtasks on crowdsourcing platforms. In this paper, we reveal the significant role of work environments in the shaping of crowd work. First, through a pilot study surveying the good and bad experiences workers had with UI elements in crowd work, we revealed the typical issues workers face. Based on these findings, we then deployed over 100 distinct microtasks on CrowdFlower, addressing workers in India and USA in two identical batches. These tasks emulate the good and bad UI element designs that characterize crowdsourcing microtasks. We recorded hardware specifics such as CPU speed and device type, apart from software specifics including the browsers used to complete tasks, operating systems on the device, and other properties that define the work environments of crowd workers. Our findings indicate that crowd workers are embedded in a variety of work environments which influence the quality of work produced. To confirm and validate our data-driven findings we then carried out semi-structured interviews with a sample of Indian and American crowd workers from this platform. Depending on the design of UI elements in microtasks, we found that some work environments support crowd workers more than others. Based on our overall findings resulting from all the three studies, we introduce ModOp, a tool that helps to design crowdsourcing microtasks that are suitable for diverse crowd work environments. We empirically show that the use of ModOp results in reducing the cognitive load of workers, thereby improving their user experience without effecting the accuracy or task completion time.

The Life Course: An interdisciplinary framework for broadening the scope of research on crowdwork

“…• Sequential QUAN -> QUAL design studies that have both paradigms equally central to the research design, for example a longitudinal survey, followed by interviews and ethnographic observation, coupled with a geographic mapping task to determine crowdworkers' location (Kingsley et al, 2015). • Quantitatively driven, sequential intervention research using QUAN -> qual -> intervention, for example questionnaire survey and interviews with crowdworkers followed by an intervention and evaluation of it (Gadiraju et al, 2017;Khanna et al, 2010;Wang et al, 2017).…”

“…About two-thirds of the mixed-method studies in our sample were conducted within microwork platforms, predominantly MTurk, with a limited number of studies focusing on platforms such as Microsoft UHRS, LeadGenius, Amara.org or Figure Eight (Gadiraju et al, 2017;Gray et al, 2016).…”

“…In the majority of mixedmethod studies, a broad spectrum of crowdworkers was included, however some studies drew on special groups of participants such as workers from specific countries, e.g. US and/or India (Gadiraju et al, 2017;Gray et al, 2016;Khanna et al, 2010;Kingsley et al, 2015); or blind, lowincome crowdworkers (Vashistha et al, 2018). In most mixed-method studies, the sample size ranged from 100 to 502 participants; only one study drew on a significantly larger sample of 6338 participants (Gray et al, 2016), whilst three of the 16 mixed-method studies used significantly smaller samples of 22, 32 and 39 participants (Gerber & Krzywdzinski, 2019;Retelny et al, 2017;Vashistha et al, 2018).…”

“…30 Williams et al (2019) x x 31 Yin et al (2018) x x 32 Rani & Furrer (2019) x x 33 Sutherland et al (2019) x x 34 Sannon et al (2019) x x 35 d' Eon et al (2019) x x 36 Kinder et al (2019) x x 37 Morris et al (2013) x 38 Schörpf et al (2017) x 39 Dalle et al (2017) x 40 Pongratz (2018) x 41 Ho et al (2015) x 42 Hata et al (2016) x 43 Jain et al (2017) x 44 Naderi (2018) x 45 Graham et al (2017) x 46 Gadiraju et al (2017) x 47 Hirth et al (2011) x 48 Khanna et al (2010) x 49 Jacques et al (2019) x 50 Gerber & Krzywdzinski (2019) x 51 Rani & Furrer (2020) x 52 Newlands & Lutz (2020) x 53 Zyskowski et al (2015) x 54 Gadiraju et al (2015) x 55 Barnes et al (2015) x 56 D' Cruz & Noronha (2016) x 57 Gadiraju et al (2016) x 58 Kost et al (2018) x 59 Posch et al (2018) x 60 Ludec et al (2019) x 61 Margaryan (2019a) x 62 Bucher et al (2019) x 63 Chen et al (2019) x 64 Zhuang & Gadiraju (2019) x 65 Margaryan (2019b) x 66 Gadiraju & Demartini (2019) x 67 Gadiraju et al (2018a) x 68 Ho et al (2018) x 69 Ma et al (2018) x 70 Mubarak et al (2016) x…”

The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurk

“…HITs are also known as microtasks, which include identifying objects from images or videos, answering survey questions, etc. More often than not, crowdworkers earn below the minimum wage (Hara et al, 2018), which has been associated with illdesigned tasks and high rejection rates of completed tasks, among other factors (Gadiraju et al, 2017;McInnis et al, 2016). At the same time, crowdworkers find themselves in intense competition against each other: to identify high-paying microtasks before other crowdworkers to secure them, and then complete them as soon as possible and to the best of their abilities so that they can move on to the next batch of microtasks (Gerber, 2021), and ultimately increase their overall ratings and income.…”