Are BERTs Sensitive to Native Interference in L2 Production?

Tang, Zixin; Mitra, Prasenjit; Reitter, David

doi:10.18653/v1/2021.insights-1.6

Cited by 2 publications

(1 citation statement)

References 19 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As mentioned in Section 3, we were initially inspired by the fact that adding more training data did not seem to improve classification performance. In addition, earlier work indicated that classifying the country of origin of an author based on their English text provides good results, with Tang et al (2021) reporting an accuracy of 87% on all of IC-NALE for this task. We argue that this points at signals of L1 in the English learner texts that a classifier can pick up on, and that consequently, finding a way to make input text more homogeneous to a classifier through debiasing (Section 4.1) can lead to CEFR classification performance gains (Section 4.2).…”

Section: Methods and Resultsmentioning

confidence: 88%

Mitigating Learnerese Effects for CEFR Classification

Jalota¹,

Bourgonje²,

Sas³

et al. 2022

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

View full text Add to dashboard Cite

The role of an author's L1 in SLA can be challenging for automated CEFR classification, in that texts from different L1 groups may be too heterogeneous to combine them as training data. We experiment with recent debiasing approaches by attempting to devoid textual representations of L1 features. This results in a more homogeneous group when aggregating CEFRannotated texts from different L1 groups, leading to better classification performance. Using iterative null-space projection, we marginally improve classification performance for a linear classifier by 1 point. An MLP (e.g. non-linear) classifier remains unaffected by this procedure. We discuss possible directions of future work to attempt to increase this performance gain.

show abstract

Section: Methods and Resultsmentioning

confidence: 88%

Mitigating Learnerese Effects for CEFR Classification

Jalota¹,

Bourgonje²,

Sas³

et al. 2022

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

View full text Add to dashboard Cite

show abstract

Regional Bias in Monolingual English Language Models

Lyu,

Dost,

Koh

et al. 2024

Preprint

View full text Add to dashboard Cite

In Natural Language Processing (NLP), pre-trained language models (LLMs) are widely employed and refined for various tasks. These models have shown considerable social and geographic biases creating skewed or even unfair representations of certain groups.Research focuses on biases toward L2 (English as a second language) regions but neglects bias within L1 (first language) regions.In this work, we ask if there is regional bias within L1 regions already inherent in pre-trained LLMs and, if so, what the consequences are in terms of downstream model performance.We contribute an investigation framework specifically tailored for low-resource regions, offering a method to identify bias without imposing strict requirements for labeled datasets. Our research reveals subtle geographic variations in the word embeddings of BERT, even in cultures traditionally perceived as similar. These nuanced features, once captured, have the potential to significantly impact downstream tasks. Generally, models exhibit comparable performance on datasets that share similarities, and conversely, performance may diverge when datasets differ in their nuanced features embedded within the language. It is crucial to note that estimating model performance solely based on standard benchmark datasets may not necessarily apply to the datasets with distinct features from the benchmark datasets. Our proposed framework plays a pivotal role in identifying and addressing biases detected in word embeddings, particularly evident in low-resource regions such as New Zealand.

show abstract

Are BERTs Sensitive to Native Interference in L2 Production?

Cited by 2 publications

References 19 publications

Mitigating Learnerese Effects for CEFR Classification

Mitigating Learnerese Effects for CEFR Classification

Regional Bias in Monolingual English Language Models

Contact Info

Product

Resources

About