Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

Shmueli, Boaz; Fell, Jan; Ray, Soumya; Ku, Lun-Wei

doi:10.18653/v1/2021.naacl-main.295

Cited by 40 publications

(28 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There has been work on proposing guidelines for requesters (Sabou et al, 2014), incorporating workers into the IRB process (Libuše Hannah Vepřek, 1 Code for our experiments is attached to this paper arXiv:2105.12762v1 [cs.CL] 26 May 2021 2020), and developing tools to help workers address the power imbalance in the online workplace Silberman, 2013, 2016). Concurrent with this work, another study showed that crowdsourcing is being used more each year in NLP research, and there is limited awareness of the ethical issues in this type of work (Shmueli et al, 2021).…”

Section: Background and Related Workmentioning

confidence: 86%

Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing

Kummerfeld¹

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Extensive work has argued in favour of paying crowd workers a wage that is at least equivalent to the U.S. federal minimum wage. Meanwhile, research on collecting high quality annotations suggests using a qualification that requires workers to have previously completed a certain number of tasks. If most requesters who pay fairly require workers to have completed a large number of tasks already then workers need to complete a substantial amount of poorly paid work before they can earn a fair wage. Through analysis of worker discussions and guidance for researchers, we estimate that workers spend approximately 2.25 months of full time effort on poorly paid tasks in order to get the qualifications needed for better paid tasks. We discuss alternatives to this qualification and conduct a study of the correlation between qualifications and work quality on two NLP tasks. We find that it is possible to reduce the burden on workers while still collecting high quality data.

show abstract

Section: Background and Related Workmentioning

confidence: 86%

Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing

Kummerfeld¹

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Research has also been conducted to investigate annotation bias and annotator pools (Al Kuwatly et al, 2020;Waseem, 2016;Ross et al, 2017;Shmueli et al, 2021;Posch et al, 2018), as well as bias (especially racial) in existing datasets (Davidson et al, 2019b;Laugier et al, 2021). It was found that data can reflect and propagate annotator bias.…”

Section: Related Workmentioning

confidence: 99%

Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments

Maronikolakis¹,

Wisiorek²,

Nann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Building on current work on multilingual hate speech (e.g., Ousidhoum et al. (2019)) and hate speech reduction (e.g., Sap et al. ( 2020)), we present XTREMESPEECH, 1 a new hate speech dataset containing 20,297 social media passages from Brazil, Germany, India and Kenya. The key novelty is that we directly involve the affected communities in collecting and annotating the data -as opposed to giving companies and governments control over defining and combatting hate speech. This inclusive approach results in datasets more representative of actually occurring online speech and is likely to facilitate the removal of the social media content that marginalized communities view as causing the most harm. Based on XTREMESPEECH, we establish novel tasks with accompanying baselines, provide evidence that cross-country training is generally not feasible due to cultural differences between countries and perform an interpretability analysis of BERT's predictions.

show abstract

“…(1) There is a lot of variation in the researcher perception of when ethics review is needed (Shmueli et al, 2021). Santy et al (2021) showed that less than 0.8% of NLP studies published since 2006 have sought IRB approval, and that was mostly for data collection or annotation.…”

Section: Necessary For the Following Reasonsmentioning

confidence: 99%

‘Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP

Rogers¹,

Baldwin²,

Leins³

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

A key part of the NLP ethics movement is responsible use of data, but exactly what that means or how it can be best achieved remain unclear. This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them.We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research across the community. Our proposal aims to contribute to the development of a consistent standard for data (re-)use, embraced across NLP conferences.

show abstract

Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

Cited by 40 publications

References 34 publications

Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing

Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing

Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments

‘Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP

Contact Info

Product

Resources

About