Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining

Abstract

Advances in neural modeling have achieved state-of-the-art (SOTA) results on public natural language processing (NLP) benchmarks, at times surpassing human performance. However, there is a gap between public benchmarks and real-world applications where noise, such as typographical or grammatical mistakes, is abundant and can result in degraded performance. Unfortunately, works which evaluate the robustness of neural models on noisy data and propose improvements, are limited to the English language. Upon analyzing noise in different languages, we observe that noise types vary greatly across languages. Thus, existing investigations do not generalize trivially to multilingual settings. To benchmark the performance of pretrained multilingual language models, we construct noisy datasets covering five languages and four NLP tasks and observe a clear gap in the performance between clean and noisy data in the zero-shot cross-lingual setting. After investigating several ways to boost the robustness of multilingual models in this setting, we propose Robust Contrastive Pretraining (RCP). RCP combines data augmentation with a contrastive loss term at the pretraining stage and achieves large improvements on noisy (& original test data) across two sentencelevel (+3.2%) and two sequence-labeling (+10 F1-score) multilingual classification tasks.

Zero-Shot Cross-Lingual Sentiment Classification under Distribution Shift: an Exploratory Study

“…The limited research on the robustness of multilingual models has primarily focused on being robust against specific types of noise, e.g., adversarial perturbations for Japanese Natural Language Inference (Yanaka and Mineshima, 2021), a combination of general and task-specific text transformations based on manipulating synonyms, antonyms, syntax, etc. , and introducing errors and noise through Wikipedia edits (Cooper Stickland et al, 2023). Unlike these works, we will evaluate how well zero-shot cross-lingual transfer from English to non-English test samples can generalize in scenarios where there is a shift in domain from train to test data: the domain-specific features of test samples may change, whereas the semantic sentiment features remain invariant.…”

Knowledge graph informed fake news classification via heterogeneous representation ensembles

“…Zhao et al (2021) study selection of instances in a few-shot scenario and show that methods are sensitive to the quality of the annotated data. Recently, Cooper Stickland et al (2023) proposed an effective pretraining strategy based on modeling typological, grammatical, or morphological noise in the data that boosts the cross-lingual zero-shot performance.…”