Paid Crowdsourcing, Low Income Contributors, and Subjectivity

Haralabopoulos, Giannis; Wagner, Christian; McAuley, Derek; Anagnostopoulos, Ioannis

doi:10.1007/978-3-030-19909-8_20

Cited by 11 publications

(9 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All the models presented, and in extent their ensembles, can be further improved by a range of techniques. Test augmentation [70], hyperparameter optimization [71], bias reduction [72] and tailored emotional embeddings [4,73] are some techniques that could further improve the generalization capabilities of our networks. However, the computational load over multiple iterations is extensive, as the most complex models required hours of training per epoch and dataset.…”

Section: Resultsmentioning

confidence: 99%

Ensemble Deep Learning for Multilabel Binary Classification of User-Generated Content

2020

Self Cite

View full text Add to dashboard Cite

Sentiment analysis usually refers to the analysis of human-generated content via a polarity filter. Affective computing deals with the exact emotions conveyed through information. Emotional information most frequently cannot be accurately described by a single emotion class. Multilabel classifiers can categorize human-generated content in multiple emotional classes. Ensemble learning can improve the statistical, computational and representation aspects of such classifiers. We present a baseline stacked ensemble and propose a weighted ensemble. Our proposed weighted ensemble can use multiple classifiers to improve classification results without hyperparameter tuning or data overfitting. We evaluate our ensemble models with two datasets. The first dataset is from Semeval2018-Task 1 and contains almost 7000 Tweets, labeled with 11 sentiment classes. The second dataset is the Toxic Comment Dataset with more than 150,000 comments, labeled with six different levels of abuse or harassment. Our results suggest that ensemble learning improves classification results by 1.5 % to 5.4 % .

show abstract

Section: Resultsmentioning

confidence: 99%

Ensemble Deep Learning for Multilabel Binary Classification of User-Generated Content

2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Contributors were able to only contribute in one of the tasks, and were excluded from the other three tasks. The use of external crowdsourcing eliminates biases that exist in an internal crowdsourcing task [18]. We ask contributors a simple question "What is the dominant emotion/colour?".…”

Section: Methodsmentioning

confidence: 99%

Privacy-Aware Crowd Labelling for Machine Learning Tasks

Haralabopoulos¹,

Anagnostopoulos²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The extensive use of online social media has highlighted the importance of privacy in the digital space. As more scientists analyse the data created in these platforms, privacy concerns have extended to data usage within the academia. Although text analysis is a well documented topic in academic literature with a multitude of applications, ensuring privacy of user-generated content has been overlooked. Most sentiment analysis methods require emotion labels, which can be obtained through crowdsourcing, where non-expert individuals contribute to scientific tasks. The text itself has to be exposed to third parties in order to be labelled. In an effort to reduce the exposure of online users' information, we propose a privacy preserving text labelling method for varying applications, based in crowdsourcing. We transform text with different levels of privacy, and analyse the effectiveness of the transformation with regards to label correlation and consistency. Our results suggest that privacy can be implemented in labelling, retaining the annotational diversity and subjectivity of traditional labelling.

show abstract

“…Contributors were able to only contribute in one of the tasks, and were excluded from the other three tasks. The use of external crowdsourcing eliminates biases that exist in an internal crowdsourcing task [10]. We ask contributors a simple question "What is the dominant emotion/colour?".…”

Section: Methodsmentioning

confidence: 99%

Privacy-Preserving Text Labelling Through Crowdsourcing

Haralabopoulos

Torres

Anagnostopoulos

et al. 2021

Artificial Intelligence Applications and Innovations. AIAI 2021 IFIP WG 12.5 International Workshops

Self Cite

View full text Add to dashboard Cite

The extensive use of online social media has highlighted the importance of privacy in the digital space. As more scientists analyse the data created in these platforms, privacy concerns have extended to data usage within the academia. Although text analysis is a well documented topic in academic literature with a multitude of applications, ensuring privacy of user-generated content has been overlooked. In an effort to reduce the exposure of online users' information, we propose a privacy-preserving text labelling method for varying applications, based in crowdsourcing. We transform text with different levels of privacy and analyse the effectiveness of the transformation with regards to label correlation. To demonstrate the adaptive nature of our approach we also employ a TF/IDF filtering transformation. Our results suggest that total privacy can be implemented in labelling, retaining the annotational diversity and subjectivity of traditional labelling. The privacy-preserving labelling, with the use of NRC lexicon, demonstrates an average 0.11 Mean Spearman's Rho correlation, boosted to 0.124 with TF/IDF filtering.

show abstract

Paid Crowdsourcing, Low Income Contributors, and Subjectivity

Cited by 11 publications

References 11 publications

Ensemble Deep Learning for Multilabel Binary Classification of User-Generated Content

Ensemble Deep Learning for Multilabel Binary Classification of User-Generated Content

Privacy-Aware Crowd Labelling for Machine Learning Tasks

Privacy-Preserving Text Labelling Through Crowdsourcing

Contact Info

Product

Resources

About