Privacy-Preserving Text Labelling Through Crowdsourcing

Abstract

The extensive use of online social media has highlighted the importance of privacy in the digital space. As more scientists analyse the data created in these platforms, privacy concerns have extended to data usage within the academia. Although text analysis is a well documented topic in academic literature with a multitude of applications, ensuring privacy of user-generated content has been overlooked. In an effort to reduce the exposure of online users' information, we propose a privacy-preserving text labelling method for varying applications, based in crowdsourcing. We transform text with different levels of privacy and analyse the effectiveness of the transformation with regards to label correlation. To demonstrate the adaptive nature of our approach we also employ a TF/IDF filtering transformation. Our results suggest that total privacy can be implemented in labelling, retaining the annotational diversity and subjectivity of traditional labelling. The privacy-preserving labelling, with the use of NRC lexicon, demonstrates an average 0.11 Mean Spearman's Rho correlation, boosted to 0.124 with TF/IDF filtering.

“Sentio Ergo Est”: Unmasking the Emotional Realities of Misperception

“…Subsequently to the presentation, psychophysiological responses, such as subcutaneous sweating (SCR) and heart-rate (HR) responses, and/or participant ratings, such as ratings for valence and intensity, are assessed to explore unconscious responses to the masked targets (see for example Lo & Zeki, 2014;Minkin et al, 2019; for participant ratings format developments, see Haralabopoulos et al, 2020;Bond et al, 2021;Yu et al, 2022;Leong et al, 2023).…”

Learning emotional dialects: A British population study of cross-cultural communication

“…All methods of face presentation included the overt presentation of faces. They included participant well-being oriented interactive feedback (see Bond et al, 2021) and included catch-trial methods for providing reliably subjective unbiased responses for the recognition of emotion (Haralabopoulos et al, 2020).…”

Assessment of bias in a crowdsourcing accessibility system using serious games

“…These methods are built on the premise that workers having low expertise breaks down quality (Eickhoff, 2014;Kittur et al, 2013), depending on the existence of a definite ground truth. For this reason, they are unsuitable for subjective tasks (Haralabopoulos et al, 2020;Kairam and Heer, 2016), which may benefit from applying the notion of crowd truth (Aroyo and Welty, 2015) and take into account multiple perspectives and interpretations. Our approach was to create a serious game based on a surrogate truth gen-erated by a small group of experts, but instead of using this benchmark to evaluate the quality of individual workers, we used it to probe the existence of behavioral biases between demographic groups.…”

“…It can be financial success, romantic love, or, as the Beatles sang, maybe even a warm gun. Measuring quality in these scenarios seems to be an open problem (Kittur et al, 2013), and traditional methods like majority voting are unsuitable for subjective tasks (Haralabopoulos et al, 2020). Thanking participants and explaining the importance of requested tasks can increase workers' motivation (Chandler and Kapelner, 2013), and combining subjective work…”

“…All of these methods take one important aspect for granted: the possibility of defining what "correct" means. They are unsuitable for subjective tasks (Haralabopoulos et al, 2020) such as sentiment analysis and, as in our case, measuring accessibility. Aroyo and Welty (2015) argue that assuming there is one correct interpretation for every input example is a fallacy.…”