Evaluating and improving the usability of Mechanical Turk for low-income workers in India

Khanna, Shashank; Ratan, Aishwarya Lakshmi; Davis, James; Thies, William

doi:10.1145/1926180.1926195

Cited by 80 publications

(66 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Examples are Khanna et al (2010)'s investigation of the influence of HIT interface design on Indian workers' ability to successfully finish a HIT or Grady and Lease's (2010) study of human factors in HIT design. The interface-related study in Sect.…”

Section: Related Workmentioning

confidence: 99%

Increasing cheat robustness of crowdsourcing tasks

2012

View full text Add to dashboard Cite

Crowdsourcing successfully strives to become a widely used means of collecting large-scale scientific corpora. Many research fields, including Information Retrieval, rely on this novel way of data acquisition. However, it seems to be undermined by a significant share of workers that are primarily interested in producing quick generic answers rather than correct ones in order to optimise their time-efficiency and, in turn, earn more money. Recently, we have seen numerous sophisticated schemes of identifying such workers. Those, however, often require additional resources or introduce artificial limitations to the task. In this work, we take a different approach by investigating means of a priori making crowdsourced tasks more resistant against cheaters.

show abstract

Section: Related Workmentioning

confidence: 99%

Increasing cheat robustness of crowdsourcing tasks

2012

View full text Add to dashboard Cite

show abstract

“…Khanna et al studied usability barriers that were prevalent on AMT, which prevented workers with little digital literacy skills from participating and completing work on AMT [40]. Authors showed that the task instructions, user interface, and the workers' cultural context corresponded to key usability barriers.…”

Section: Crowd Work Context and Barriersmentioning

confidence: 99%

Modus Operandi of Crowd Workers

Gadiraju

Checco

Gupta

et al. 2017

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that of work environments; defined as the hardware and software affordances at the disposal of crowd workers which are used to complete microtasks on crowdsourcing platforms. In this paper, we reveal the significant role of work environments in the shaping of crowd work. First, through a pilot study surveying the good and bad experiences workers had with UI elements in crowd work, we revealed the typical issues workers face. Based on these findings, we then deployed over 100 distinct microtasks on CrowdFlower, addressing workers in India and USA in two identical batches. These tasks emulate the good and bad UI element designs that characterize crowdsourcing microtasks. We recorded hardware specifics such as CPU speed and device type, apart from software specifics including the browsers used to complete tasks, operating systems on the device, and other properties that define the work environments of crowd workers. Our findings indicate that crowd workers are embedded in a variety of work environments which influence the quality of work produced. To confirm and validate our data-driven findings we then carried out semi-structured interviews with a sample of Indian and American crowd workers from this platform. Depending on the design of UI elements in microtasks, we found that some work environments support crowd workers more than others. Based on our overall findings resulting from all the three studies, we introduce ModOp, a tool that helps to design crowdsourcing microtasks that are suitable for diverse crowd work environments. We empirically show that the use of ModOp results in reducing the cognitive load of workers, thereby improving their user experience without effecting the accuracy or task completion time.

show abstract

“…Platforms such as txt Eagle [27] and mClerk [28] aim to enable workers to earn supplemental income on low-end mobile phones. Others have exa mined the usage [29,30] and non-usage [31] of Mechanical Turk in India, where approximately one third of Turkers reside. Effo rts such as Ushahidi [32] and Mission 4636 in Hait i [33] have leveraged crowd workers to respond to crises in developing countries.…”

Section: Related Workmentioning

confidence: 99%