Assessing the Generalization Capacity of Pre-trained Language Models through Japanese Adversarial Natural Language Inference

Yanaka, Hitomi; Mineshima, Koji

doi:10.18653/v1/2021.blackboxnlp-1.26

Cited by 3 publications

(3 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are few works for other languages [3,16,23,34], and particularly, for Vietnamese [17]. These works however focus on constructing open-domain datasets and also only for the monolingual problem.…”

Section: Related Workmentioning

confidence: 99%

Hevenli: A Large Dataset for English-Vietnamese Cross-Lingual Natural Language Inference in Healthcare

Ngo,

To,

Nguyen

et al. 2024

Preprint

View full text Add to dashboard Cite

We introduce Hevenli, a novel large dataset for English-Vietnamese cross-lingual natural language inference learning. Different from the existing datasets which are not domain specific and diverse in style, ours is specific to healthcare domain and consists of news-style sentence pairs. Notably, our dataset is more than two times larger than the largest existing one. We conducted an exploratory analysis to examine the topics of the dataset. Also, to set the baseline results, we evaluated the state-of-the-art methods on the dataset. The obtained results show that our dataset is a challenging corpus which is potentially useful for more advanced research and developing practical applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Hevenli: A Large Dataset for English-Vietnamese Cross-Lingual Natural Language Inference in Healthcare

Ngo,

To,

Nguyen

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The limited research on the robustness of multilingual models has primarily focused on being robust against specific types of noise, e.g., adversarial perturbations for Japanese Natural Language Inference (Yanaka and Mineshima, 2021), a combination of general and task-specific text transformations based on manipulating synonyms, antonyms, syntax, etc. , and introducing errors and noise through Wikipedia edits (Cooper Stickland et al, 2023).…”

Section: Cross-lingual Transfer Under Distribution Shiftmentioning

confidence: 99%

Zero-Shot Cross-Lingual Sentiment Classification under Distribution Shift: an Exploratory Study

De Raedt,

Bitew,

Godin

et al. 2023

Proceedings of the 3rd Workshop on Multi-Lingual Representation Learning (MRL)

View full text Add to dashboard Cite

The brittleness of finetuned language model performance on out-of-distribution (OOD) test samples in unseen domains has been wellstudied for English, yet is unexplored for multilingual models. Therefore, we study generalization to OOD test data specifically in zero-shot cross-lingual transfer settings, analyzing performance impacts of both language and domain shifts between train and test data. We further assess the effectiveness of counterfactually augmented data (CAD) in improving OOD generalization for the cross-lingual setting, since CAD has been shown to benefit in a monolingual English setting. Finally, we propose two new approaches for OOD generalization that avoid the costly annotation process associated with CAD, by exploiting the power of recent large language models (LLMs). We experiment with 3 multilingual models, LaBSE, mBERT, and XLM-R trained on English IMDb movie reviews, and evaluate on OOD test sets in 13 languages: Amazon product reviews, Tweets, and Restaurant reviews. Results echo the OOD performance decline observed in the monolingual English setting. Further, (i) counterfactuals from the original high-resource language do improve OOD generalization in the low-resource language, and (ii) our newly proposed cost-effective approaches reach similar or up to to +3.1% better accuracy than CAD for Amazon and Restaurant reviews.

show abstract

“…Templates We make templates and generate sentences with them using the codebase developed by 9 We provide examples for each condition in Appendix B. Yanaka and Mineshima (2021). 10 Following are examples of templates and sentences.…”

Section: Data Generationmentioning

confidence: 99%

PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments

Asami,

Sugawara

2023

Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

What makes a presupposition of an utteranceinformation taken for granted by its speakerdifferent from other pragmatic inferences such as an entailment is projectivity (e.g., the negative sentence the boy did not stop shedding tears presupposes the boy had shed tears before). The projectivity may vary depending on the combination of presupposition triggers and environments. However, prior natural language understanding studies fail to take it into account as they either use no human baseline or include only negation as an entailment-canceling environment to evaluate models' performance. The current study attempts to reconcile these issues. We introduce a new dataset, projectivity of presupposition (PROPRES), which includes 12k premise-hypothesis pairs crossing six triggers involving some lexical variety with five environments. Our human evaluation reveals that humans exhibit variable projectivity in some cases. However, the model evaluation shows that the best-performed model, DeBERTa, does not fully capture it. Our findings suggest that probing studies on pragmatic inferences should take extra care of the human judgment variability and the combination of linguistic items.

show abstract

Assessing the Generalization Capacity of Pre-trained Language Models through Japanese Adversarial Natural Language Inference

Cited by 3 publications

References 33 publications

Hevenli: A Large Dataset for English-Vietnamese Cross-Lingual Natural Language Inference in Healthcare

Hevenli: A Large Dataset for English-Vietnamese Cross-Lingual Natural Language Inference in Healthcare

Zero-Shot Cross-Lingual Sentiment Classification under Distribution Shift: an Exploratory Study

PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments

Contact Info

Product

Resources

About