EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

Ma, Weicheng; Datta, Samiha; Wang, Lili; Vosoughi, Soroush

doi:10.18653/v1/2022.findings-acl.221

Cited by 5 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, following existing research (Mora, 2013;Tomlinson et al, 2014;Hershcovich et al, 2022), we define culture as the combination of human beliefs, norms, and customs among groups. Previous work in natural language processing (NLP) has primarily focused on cultural investigation of models (Hutchinson et al, 2020;Ross et al, 2021;Ma et al, 2022), with little emphasis on dialogue agents. Besides, probing is a popular way to study the characteristics of models or agents (Hämmerl et al, 2022;Arora et al, 2022;.…”

Section: Discussionmentioning

confidence: 99%

Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)

2023

View full text Add to dashboard Cite

This paper investigates the performance of massively multilingual neural machine translation (NMT) systems in translating Yorùbá greetings (ε kú 1 ), which are a big part of Yorùbá language and culture, into English. To evaluate these models, we present IkiniYorùbá, a Yorùbá-English translation dataset containing some Yorùbá greetings, and sample use cases. We analysed the performance of different multilingual NMT systems including Google Translate and NLLB and show that these models struggle to accurately translate Yorùbá greetings into English. In addition, we trained a Yorùbá-English model by finetuning an existing NMT model on the training split of IkiniYorùbá and this achieved better performance when compared to the pre-trained multilingual NMT models, although they were trained on a large volume of data. * Equal contribution. 1 For simplicity of notation in the title, we make use of εthe Beninese Yorùbá letter representation of E . (which is used in Nigeria), and provides the context of greeting.Source: E . kú ojúmó . , e . sì kú déédé àsìkò yìí. Target: Good morning and compliment for this period.NLLB: You have died, and you have died to this hour. Google Translate: Die every day, and die at this time. Our Model: Good morning and compliment for this time.

show abstract

Section: Discussionmentioning

confidence: 99%

Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)

2023

View full text Add to dashboard Cite

show abstract

“…They investigate the regional differences at the performance level, while this paper tries to illustrate bias at the intrinsic level, i.e., differences in embedding space. Nevertheless, research by Ma et al (2022) points to bias in resourceabundant regions. This paper strives to diminish dependence on labeled task data by examining regional bias at the word embedding level before the fine-tuning process, potentially impacting downstream task performance.…”

Section: Studies Investigating Regional Differences In Llmsmentioning

confidence: 99%

“…Language is intertwined with culture due to differences in word usage behavior (Loveys et al, 2018), writing styles (Ma et al, 2022), common sense knowledge, debatable topics, and value systems (Hershcovich et al, 2022). Research has shown that these demographic differences in the task domain will harm the performance of downstream Natural Language Processing (NLP) tasks (Ma et al, 2022;Ghosh et al, 2021;Sun et al, 2021;González et al, 2020;Tan et al, 2020;Loveys et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

“…Particularly, Zhang et al (2021) show that most of the word embeddings reflect more of the language habits of European-educated males, neglecting other subsets of the population. This constitutes a biased selection of the population (Hershcovich et al, 2022;Ma et al, 2022) and raises concerns about the non-selected groups' representation within the dataset (Hershcovich et al, 2022;Wolfe and Caliskan, 2021), which will probably cause harms in applications. (Ghosh et al, 2021;Tan et al, 2020;González et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

“…While research mostly focuses on cross-culture problems in cross-lingual language models, the monolingual model -English in this case -may not be free of culture difference bias either (Hershcovich et al, 2022). Recent studies on English language models focus on the geographic influence of non-traditional English-speaking (L2) regions (Tan et al, 2020;Ghosh et al, 2021), e.g., researchers discovered differences in emotional responses Ghosh et al (2021) and word usage Ma et al (2022) between L2 regions. Others investigate bias in models on NLP tasks for different social groups within one region (Zhang et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Regional Bias in Monolingual English Language Models

Lyu,

Dost,

Koh

et al. 2024

Preprint

View full text Add to dashboard Cite

In Natural Language Processing (NLP), pre-trained language models (LLMs) are widely employed and refined for various tasks. These models have shown considerable social and geographic biases creating skewed or even unfair representations of certain groups.Research focuses on biases toward L2 (English as a second language) regions but neglects bias within L1 (first language) regions.In this work, we ask if there is regional bias within L1 regions already inherent in pre-trained LLMs and, if so, what the consequences are in terms of downstream model performance.We contribute an investigation framework specifically tailored for low-resource regions, offering a method to identify bias without imposing strict requirements for labeled datasets. Our research reveals subtle geographic variations in the word embeddings of BERT, even in cultures traditionally perceived as similar. These nuanced features, once captured, have the potential to significantly impact downstream tasks. Generally, models exhibit comparable performance on datasets that share similarities, and conversely, performance may diverge when datasets differ in their nuanced features embedded within the language. It is crucial to note that estimating model performance solely based on standard benchmark datasets may not necessarily apply to the datasets with distinct features from the benchmark datasets. Our proposed framework plays a pivotal role in identifying and addressing biases detected in word embeddings, particularly evident in low-resource regions such as New Zealand.

show abstract

Regional bias in monolingual English language models

Lyu,

Dost,

Koh

et al. 2024

Mach Learn

View full text Add to dashboard Cite

In Natural Language Processing (NLP), pre-trained language models (LLMs) are widely employed and refined for various tasks. These models have shown considerable social and geographic biases creating skewed or even unfair representations of certain groups. Research focuses on biases toward L2 (English as a second language) regions but neglects bias within L1 (first language) regions. In this work, we ask if there is regional bias within L1 regions already inherent in pre-trained LLMs and, if so, what the consequences are in terms of downstream model performance. We contribute an investigation framework specifically tailored for low-resource regions, offering a method to identify bias without imposing strict requirements for labeled datasets. Our research reveals subtle geographic variations in the word embeddings of BERT, even in cultures traditionally perceived as similar. These nuanced features, once captured, have the potential to significantly impact downstream tasks. Generally, models exhibit comparable performance on datasets that share similarities, and conversely, performance may diverge when datasets differ in their nuanced features embedded within the language. It is crucial to note that estimating model performance solely based on standard benchmark datasets may not necessarily apply to the datasets with distinct features from the benchmark datasets. Our proposed framework plays a pivotal role in identifying and addressing biases detected in word embeddings, particularly evident in low-resource regions such as New Zealand.

show abstract

EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

Cited by 5 publications

References 16 publications

Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)

Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)

Regional Bias in Monolingual English Language Models

Regional bias in monolingual English language models

Contact Info

Product

Resources

About