A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

Suárez, Pedro Javier Ortiz; Romary, Laurent; Sagot, Benoît

doi:10.18653/v1/2020.acl-main.156

Cited by 66 publications

(31 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We sample 10M EN-RU sentences from the WMT'19 shared task (Ma et al, 2019), and 80M RU sentences from the CoNLL'17 shared task to train embeddings. To simulate lowresource scenarios, we sample 10K, 100K and 1M UK sentences from the CoNLL'17 shared task and BE sentences from the OSCAR corpus (Ortiz Suárez et al, 2020). We use TED dev/test sets for both languages pairs (Cettolo et al, 2012).…”

Section: Methodsmentioning

confidence: 99%

Machine Translation into Low-resource Language Varieties

Kumar¹,

Anastasopoulos²,

Wintner³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

State-of-the-art machine translation (MT) systems are typically trained to generate "standard" target language; however, many languages have multiple varieties (regional varieties, dialects, sociolects, non-native varieties) that are different from the standard language. Such varieties are often low-resource, and hence do not benefit from contemporary NLP solutions, MT included. We propose a general framework to rapidly adapt MT systems to generate language varieties that are close to, but different from, the standard target language, using no parallel (sourcevariety) data. This also includes adaptation of MT systems to low-resource typologicallyrelated target languages. 1 We experiment with adapting an English-Russian MT system to generate Ukrainian and Belarusian, an English-Norwegian Bokmål system to generate Nynorsk, and an English-Arabic system to generate four Arabic dialects, obtaining significant improvements over competitive baselines.

show abstract

Section: Methodsmentioning

confidence: 99%

Machine Translation into Low-resource Language Varieties

Kumar¹,

Anastasopoulos²,

Wintner³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…We fine-tune three adaptations of BERT (Devlin et al, 2019): mBERT, trained by the original authors on a corpus consisting of the entire Wikipedia dumps of 100 languages; HeBERT (Chriqui and Yahav, 2021), trained on the OSCAR corpus (Ortiz Suárez et al, 2020) and Hebrew Wikipedia; AlephBERT (Seker et al, 2021), also trained on the OSCAR corpus, with an additional 71.5 million tweets in Hebrew. All models are equivalent in size to BERT-base, i.e.…”

Section: Experiments Setupmentioning

confidence: 99%

ParaShoot: A Hebrew Question Answering Dataset

Keren¹,

Levy²

2021

Proceedings of the 3rd Workshop on Machine Reading for Question Answering

View full text Add to dashboard Cite

NLP research in Hebrew has largely focused on morphology and syntax, where rich annotated datasets in the spirit of Universal Dependencies are available. Semantic datasets, however, are in short supply, hindering crucial advances in the development of NLP technology in Hebrew. In this work, we present PARASHOOT, the first question answering dataset in modern Hebrew. The dataset follows the format and crowdsourcing methodology of SQuAD, and contains approximately 3000 annotated examples, similar to other questionanswering datasets in low-resource languages. We provide the first baseline results using recently-released BERT-style models for Hebrew, showing that there is significant room for improvement on this task.

show abstract

“…NASCA was pretrained with the documents of the Catalan training set of the DACSA corpus (including some documents discarded in the corpora creation process [15]), the Catalan subset of the OSCAR corpus [31], and the dump from 20 April 2021 of the Catalan version of the Wikipedia. In total, 9.3 GB of raw text (2.5 millions of documents) were used to pretrain it.…”

Section: Summarization Modelsmentioning

confidence: 99%

NASca and NASes: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish

et al. 2021

View full text Add to dashboard Cite

Most of the models proposed in the literature for abstractive summarization are generally suitable for the English language but not for other languages. Multilingual models were introduced to address that language constraint, but despite their applicability being broader than that of the monolingual models, their performance is typically lower, especially for minority languages like Catalan. In this paper, we present a monolingual model for abstractive summarization of textual content in the Catalan language. The model is a Transformer encoder-decoder which is pretrained and fine-tuned specifically for the Catalan language using a corpus of newspaper articles. In the pretraining phase, we introduced several self-supervised tasks to specialize the model on the summarization task and to increase the abstractivity of the generated summaries. To study the performance of our proposal in languages with higher resources than Catalan, we replicate the model and the experimentation for the Spanish language. The usual evaluation metrics, not only the most used ROUGE measure but also other more semantic ones such as BertScore, do not allow to correctly evaluate the abstractivity of the generated summaries. In this work, we also present a new metric, called content reordering, to evaluate one of the most common characteristics of abstractive summaries, the rearrangement of the original content. We carried out an exhaustive experimentation to compare the performance of the monolingual models proposed in this work with two of the most widely used multilingual models in text summarization, mBART and mT5. The experimentation results support the quality of our monolingual models, especially considering that the multilingual models were pretrained with many more resources than those used in our models. Likewise, it is shown that the pretraining tasks helped to increase the degree of abstractivity of the generated summaries. To our knowledge, this is the first work that explores a monolingual approach for abstractive summarization both in Catalan and Spanish.

show abstract

A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

Cited by 66 publications

References 24 publications

Machine Translation into Low-resource Language Varieties

Machine Translation into Low-resource Language Varieties

ParaShoot: A Hebrew Question Answering Dataset

NASca and NASes: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish

Contact Info

Product

Resources

About