Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing

Martín, David; Carpendale, Sheelagh; Gupta, Neha; Hoßfeld, Tobias; Naderi, Babak; Redi, Judith; Siahaan, Ernestasia; Wechsung, Ina

doi:10.1007/978-3-319-66435-4_3

Cited by 19 publications

(16 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…While the distribution of the countries of residence for Microworkers differs from previous studies [12], the United States and India are reported as the most active workers countries on MTurk [5]. The share of male participants is similar to previously reported values from other studies for MTurk and Microworkers [5,11,23].…”

Section: Demographicssupporting

confidence: 78%

“…The analysis of the self-reported age of the participants shows a median age of 30 years for sub-study A and C, while the participants of sub-study B are little younger with a median of 27 years, and the participants of sub-study D are little older with a median of 31 years. Again, the age distribution is similar to the results for other studies [5,11,23]. As the age may have an impact on the accuracy of the workers' outcome [18], we statistically analyze the age…”

Section: Demographicssupporting

confidence: 71%

See 1 more Smart Citation

In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE?

et al. 2020

Self Cite

View full text Add to dashboard Cite

Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.

show abstract

Section: Demographicssupporting

confidence: 78%

Section: Demographicssupporting

confidence: 71%

In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE?

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…small and repetitive tasks which can be performed quickly, such as writing small texts, recognizing pictures, providing transcriptions or verifying the accuracy of data. Several surveys are aimed at better understanding the demographic characteristics of users on online labor platforms (Ipeirotis, 2010;Ross et al, 2010;Martin et al, 2017). Martin et al (2017) discuss that most demographic studies on MTurk find the majority of workers to be U.S. based (50%-60%).…”

Section: Online Labor Platforms and Microtaskingmentioning

confidence: 99%

“…Several surveys are aimed at better understanding the demographic characteristics of users on online labor platforms (Ipeirotis, 2010;Ross et al, 2010;Martin et al, 2017). Martin et al (2017) discuss that most demographic studies on MTurk find the majority of workers to be U.S. based (50%-60%). Based on their own survey conducted on the platforms MTurk, Crowdee and Microworkers.com, and in line with other studies on the topic, the authors show that most crowdworkers across platforms exhibit a high eduction level with a with a college or advanced degree and that online labor platforms tend to be dominated by male workers.…”

Section: Online Labor Platforms and Microtaskingmentioning

confidence: 99%

Unemployment and Online Labor

Borchert

Hirth²,

Kummer

et al. 2018

SSRN Journal

View full text Add to dashboard Cite

Online labor markets experienced a rapid growth in recent years. They allow for long-distance transactions and offer workers access to a potentially 'global' pool of labor demand. As such, they bear the potential to act as a substitute for shrinking local income opportunities. Using detailed U.S. data from a large online labor platform for microtasks, we study how local unemployment affects participation and work intensity online. We find that, at the extensive margin, an increase in commuting zone level unemployment is associated with more individuals joining the platform and becoming active in fulfilling tasks. At the intensive margin, our results show that with higher unemployment rates, online labor supply becomes more elastic. These results are driven by a decrease of the reservation wage during standard working hours. Finally, the effects are transient and do not translate to a permanent increase in platform participation by incumbent users. Our findings highlight that many workers consider online labor markets as a substitute to offline work for generating income, especially in periods of low local labor demand. However, the evidence also suggests that, despite their potential to attract workers, online markets for microtasks are currently not viable as a long run alternative for most workers.

show abstract

“…In theory, the lack of formal working hours on platforms like AMT (and broader 'gig' work in general) allows people to decide when they work, how much work they do and to enjoy complete freedom to choose where they work [13]. The majority of AMT workers complete tasks from their homes [52], but they also complete tasks on the go, such as on their mobile phone, in internet cafés or between classes [24,52].…”

Section: Introductionmentioning

confidence: 99%

Monotasking or Multitasking

Lascau

Gould

Cox

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document. When citing, please reference the published version. Take down policy While the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has been uploaded in error or has been deemed to be commercially or otherwise sensitive.

show abstract

Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing

Cited by 19 publications

References 30 publications

In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE?

In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE?

Unemployment and Online Labor

Monotasking or Multitasking

Contact Info

Product

Resources

About