Disembodied Machine Learning: On the Illusion of Objectivity in NLP

Waseem, Zeerak; Lulz, Smarika; Bingel, Joachim; Augenstein, Isabelle

doi:10.48550/arxiv.2101.11974

Cited by 7 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regarding objectivity, neither the LENA ® algorithm nor any other algorithm can be claimed to be objective. Biases arise throughout the various stages of algorithm development, including during data collection, manual annotation, and algorithm design -see Waseem et al (2021) for a discussion on the illusion of objectivity in ML algorithms. Acknowledging the limitations of algorithms, particularly machine learning algorithms, is crucial if one wants to start documenting biases and building more inclusive algorithms.…”

Section: Discussionmentioning

confidence: 99%

Artificial neural networks to analyze and simulate language acquisition in children

Lavechin

2023

Preprint

View full text Add to dashboard Cite

Lightweight child-worn recorders that collect audio across an entire day allow for a big-data approach to the study of language development. By collecting the child's production and linguistic environment, these recordings offer us a uniquely naturalistic view of everyday language uses. However, such recordings quickly accumulate thousands of hours of audio and require the use of automatic speech processing algorithms. Besides providing ecologically-valid measures of what children hear and say, these recordings can fuel computational models of early language acquisition with what infants truly hear. This opens up new opportunities for running realistic language learning simulations.A first aspect of my doctoral work is dedicated to developing automatic speech processing algorithms for child-centered long-form recordings. In this manuscript, I first show that current state-of-the-art automatic speech recognition systems fail to capture the complexity of naturalistic speech as found in long-forms. I then present our attempt to propose a free, open-source, and more accurate alternative to the LENA proprietary software, which is currently the standard tool for obtaining automatic analyses of long-forms. Using supervised learning methods, my collaborators and I built a suite of speech processing tools to detect voice activity, identify voice signal sources (child vocalizations, female or male speech), count the number of linguistic units (phonemes, syllables, or words), and estimate the quantity of background noise and reverberation. A second aspect of my doctoral work is dedicated to computational models of early language acquisition. I present a first modeling study showing that self-supervised learning algorithms trained on audiobooks can learn phonetic and lexical aspects of their training language. I then show that the same algorithm trained on ecological long-forms needs inductive biases to learn phonetic aspects of its training language reliably and reflect on whether similar inductive biases may guide language learning in infants. Interestingly, there is no evidence for lexical learning on long-forms, contrary to what has been shown in the literature on more curated data. This series of studies illustrates the importance of considering ecologically-valid input data when modeling language acquisition.

show abstract

Section: Discussionmentioning

confidence: 99%

Artificial neural networks to analyze and simulate language acquisition in children

Lavechin

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Having access to noisy sentiment-based stance data in the same languages helps, but transferring knowledge from a resource-rich language (e.g., English) from the same task (or set of task definitions) is even more beneficial, in contrast to the data's (see Section 4.1) and label's language (see Appendix C.5). Moreover, when using noisy labels from an external model, there is always a risk of introducing additional bias due to the training data and to discrepancies in the task definition (Waseem et al 2021;Bender et al 2021). We observed this for both the dast and the rustance datasets.…”

Section: Discussionmentioning

confidence: 99%

Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-training

Hardalov

Arora

Nakov

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

The goal of stance detection is to determine the viewpoint expressed in a piece of text towards a target. These viewpoints or contexts are often expressed in many different languages depending on the user and the platform, which can be a local news outlet, a social media platform, a news forum, etc. Most research on stance detection, however, has been limited to working with a single language and on a few limited targets, with little work on cross-lingual stance detection. Moreover, non-English sources of labelled data are often scarce and present additional challenges. Recently, large multilingual language models have substantially improved the performance on many non-English tasks, especially such with a limited number of examples. This highlights the importance of model pre-training and its ability to learn from few examples. In this paper, we present the most comprehensive study of cross-lingual stance detection to date: we experiment with 15 diverse datasets in 12 languages from 6 language families, and with 6 low-resource evaluation settings each. For our experiments, we build on pattern-exploiting training (PET), proposing the addition of a novel label encoder to simplify the verbalisation procedure. We further propose sentiment-based generation of stance data for pre-training, which shows sizeable improvement of more than 6% F1 absolute in few-shot learning settings compared to several strong baselines.

show abstract

“…The well-established GLUE-style benchmarks evaluate systems using mean aggregation over heterogeneous task-specific metrics (Wang et al, 2018(Wang et al, , 2019(Wang et al, , 2021a. Based on the criticism of this evaluation protocol by the research community (e.g., Waseem et al, 2021;Mishra and Arunkumar, 2021;Agarwal et al, 2021), we recognize that mean aggregation in our case does not account for the nature of the adversarial transformations and attacks and task specifications, such as the task type, domain, and the number of episodes in D train and D test . Baseline evaluation.…”

Section: Limitationsmentioning

confidence: 99%

TAPE: Assessing Few-shot Russian Language Understanding

Taktasheva¹,

Shavrina²,

Fenogenova³

et al. 2022

Findings of the Association for Computational Linguistics: EMNLP 2022

View full text Add to dashboard Cite

Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes. However, this fast-growing area lacks standardized evaluation suites for non-English languages, hindering progress outside the Anglo-centric paradigm. To address this line of research, we propose TAPE (Text Attack and Perturbation Evaluation), a novel benchmark that includes six more complex NLU tasks for Russian, covering multi-hop reasoning, ethical concepts, logic and commonsense knowledge. The TAPE's design focuses on systematic zero-shot and fewshot NLU evaluation: (i) linguistic-oriented adversarial attacks and perturbations for analyzing robustness, and (ii) subpopulations for nuanced interpretation. The detailed analysis of testing the autoregressive baselines indicates that simple spelling-based perturbations affect the performance the most, while paraphrasing the input has a more negligible effect. At the same time, the results demonstrate a significant gap between the neural and human baselines for most tasks. We publicly release TAPE 1 to foster research on robust LMs that can generalize to new tasks when little to no supervision is available.

show abstract

Disembodied Machine Learning: On the Illusion of Objectivity in NLP

Cited by 7 publications

References 20 publications

Artificial neural networks to analyze and simulate language acquisition in children

Artificial neural networks to analyze and simulate language acquisition in children

Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-training

TAPE: Assessing Few-shot Russian Language Understanding

Contact Info

Product

Resources

About