Moses and the Character-Based Random Babbling Baseline: CoAStaL at AmericasNLP 2021 Shared Task

Bollmann, Marcel; Aralikatte, Rahul; Bello, Héctor Murrieta; Hershcovich, Daniel; Lhoneux, Miryam de; Søgaard, Anders

doi:10.18653/v1/2021.americasnlp-1.28

Cited by 1 publication

(1 citation statement)

References 16 publications

(7 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…most relevant system properties, are shown in Table 5: we omit the "random babbling" baseline by Bollmann et al (2021) as well as all systems trained on parts or all of the development set from this analysis. Due to the nature of the shared task we summarize here, systems are trained on different datasets and not directly comparable; we focus on general trends and leave a principled comparison of the effects and interactions of model architectures, training techniques, and datasets to future work.…”

Section: Machine Translation Resultsmentioning

confidence: 99%

AmericasNLI: Machine translation and natural language inference systems for Indigenous languages of the Americas

Kann

Ebrahimi

Mager

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Little attention has been paid to the development of human language technology for truly low-resource languages—i.e., languages with limited amounts of digitally available text data, such as Indigenous languages. However, it has been shown that pretrained multilingual models are able to perform crosslingual transfer in a zero-shot setting even for low-resource languages which are unseen during pretraining. Yet, prior work evaluating performance on unseen languages has largely been limited to shallow token-level tasks. It remains unclear if zero-shot learning of deeper semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, a natural language inference dataset covering 10 Indigenous languages of the Americas. We conduct experiments with pretrained models, exploring zero-shot learning in combination with model adaptation. Furthermore, as AmericasNLI is a multiway parallel dataset, we use it to benchmark the performance of different machine translation models for those languages. Finally, using a standard transformer model, we explore translation-based approaches for natural language inference. We find that the zero-shot performance of pretrained models without adaptation is poor for all languages in AmericasNLI, but model adaptation via continued pretraining results in improvements. All machine translation models are rather weak, but, surprisingly, translation-based approaches to natural language inference outperform all other models on that task.

show abstract

Section: Machine Translation Resultsmentioning

confidence: 99%