Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform

Lioznova, Anna; Drutsa, Alexey; Kukushkin, Vladimir; Bezzubtseva, Anastasia

doi:10.1145/3394486.3403369

Cited by 6 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the problem was that it would sometimes take workers significant time to get to some of the HITs they had "reserved" for themselves, and hence they experienced timeouts. We thus believe there is value in exploring tools [81], that based on workers' log data, can automatically learn the best amount of time that should be allocated for a given task and then recommend to requesters to use a significantly higher time window than that time to avoid timeouts and also be sympathetic with the labor practices of some workers.…”

Section: Resultsmentioning

confidence: 99%

Quantifying the Invisible Labor in Crowd Work

Toxtli,

Suri,

Savage

2021

Preprint

View full text Add to dashboard Cite

Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these markets, which ultimately reduces workers' hourly wages. Invisible labor includes finding good tasks, messaging requesters, or managing payments. However, we currently know little about how much time crowd workers actually spend on invisible labor or how much it costs them economically. To ensure a fair and equitable future for crowd work, we need to be certain that workers are being paid fairly for all of the work they do. In this paper, we conduct a field study to quantify the invisible labor in crowd work. We build a plugin to record the amount of time that 100 workers on Amazon Mechanical Turk dedicate to invisible labor while completing 40,903 tasks. If we ignore the time workers spent on invisible labor, workers' median hourly wage was $3.76. But, we estimated that crowd workers in our study spent 33% of their time daily on invisible labor, dropping their median hourly wage to $2.83. We found that the invisible labor differentially impacts workers depending on their skill level and workers' demographics. The invisible labor category that took the most time and that was also the most common revolved around workers having to manage their payments. The second most time-consuming invisible labor category involved hyper-vigilance, where workers vigilantly watched over requesters' profiles for newly posted work or vigilantly searched for labor. We hope that through our paper, the invisible labor in crowdsourcing becomes more visible, and our results help to reveal the larger implications of the continuing invisibility of labor in crowdsourcing.

show abstract

Section: Resultsmentioning

confidence: 99%

Quantifying the Invisible Labor in Crowd Work

Toxtli,

Suri,

Savage

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It triggers a subsequent ethical dispute, namely, can there be any sort of employment relationship between investigators and human subjects in academic research in the first place? If not, then scholars are not entitled to pay crowd workers a wage, which can potentially exert an undue influence on them because a wage standard is usually higher than the majority of payment rate in crowd work (Silberman et al , 2018; Saito et al , 2019; Lioznova et al , 2020). If yes, then it generates an employment obligation in crowd work-based research.…”

Section: Crowd Work’s Fundamental Problem #2: the Confusion Of Termin...mentioning

confidence: 99%

The original sin of crowd work for human subjects research

Xia

2022

JICES

View full text Add to dashboard Cite

Purpose Academic scholars have leveraged crowd work platforms such as Amazon Mechanical Turk for human subjects research for almost two decades. However, few scholars have reflected or questioned this mode of academic research. This paper aims to examine three fundamental problems of crowd work and elaborates on their lasting effects on impacting the validity and quality of human subjects research on crowd work. Design/methodology/approach` A critical analysis is conducted on the characteristics of crowd work, and three fundamental problems of crowd work since its origin were identified, namely, the position of “Human-as-a-service,” the confusion of terminology and crowd work platforms’ abdication of responsibilities. Findings This paper explains that the three identified fundamental problems of crowd work render at least two lasting problems in crowd work-based research: first, the negligence of the teleological difference between crowd work and academic research; second, the ontological schism between scholars and institutional review boards (IRBs) in their ethical concerns and practices. Originality/value This paper critiques the foundation of crowd work-based research that has become growingly popular, extolled and taken for granted. Such a critique is deficient in literature and may seem a bit peculiar. However, we hold that it is time to take research ethics seriously in crowd work because we need to introspect and question ourselves as scholars: What is our motive or ethical stance in using crowd work for human subjects research? Is it for advancing scientific knowledge, promoting crowd workers’ welfare, or predominantly for benefiting ourselves from the fast, cheap and “good” data via crowd work?

show abstract

“…We also contribute to the second discourse focusing on the growth and development of online labor markets, which is strongly connected with the call for additional research in the field of crowdworking (Greef et al 2017;Maier and Viete 2017). With the novel dataset used in this study, we aggregate information on the main motivator of crowdworkers (Kaufmann et al 2011;Goodman et al 2013;Lioznova et al 2020) and contribute to existing efforts to extend the database on online crowdworking. Furthermore, the results of our analysis could serve as a preliminary benchmark for future studies examining the wages of new online workers in the wake of global pandemics and migration movements.…”

Section: Contribution To Current Policy Debatesmentioning

confidence: 99%

Hourly Wages in Crowdworking: A Meta-Analysis

Hornuf

Vrankar

2022

Bus Inf Syst Eng

View full text Add to dashboard Cite

In the past decade, crowdworking on online labor market platforms has become an important source of income for a growing number of people worldwide. This development has led to increasing political and scholarly interest in the wages people can earn on such platforms. This study extends the literature, which is often based on a single platform, region, or category of crowdworking, through a meta-analysis of prevalent hourly wages. After a systematic literature search, the paper considers 22 primary empirical studies, including 105 wages and 76,765 data points from 22 platforms, eight different countries, and 10 years. It is found that, on average, microtasks results in an hourly wage of less than $6. This wage is significantly lower than the mean wage of online freelancers, which is roughly three times higher when not factoring in unpaid work. Hourly wages accounting for unpaid work, such as searching for tasks and communicating with requesters, tend to be significantly lower than wages not considering unpaid work. Legislators and researchers evaluating wages in crowdworking need to be aware of this bias when assessing hourly wages, given that the majority of literature does not account for the effect of unpaid work time on crowdworking wages. To foster the comparability of different research results, the article suggests that scholars consider a wage correction factor to account for unpaid work. Finally, researchers should be aware that remuneration and work processes on crowdworking platforms can systematically affect the data collection method and inclusion of unpaid work.

show abstract

Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform

Cited by 6 publications

References 19 publications

Quantifying the Invisible Labor in Crowd Work

Quantifying the Invisible Labor in Crowd Work

The original sin of crowd work for human subjects research

Hourly Wages in Crowdworking: A Meta-Analysis

Contact Info

Product

Resources

About