Multilingual is not enough: BERT for Finnish

Virtanen, Antti; Kanerva, Jenna; Luoma, Jouni; Luotolahti, Juhani; Salakoski, Tapio; Ginter, Filip; Pyysalo, Sampo

doi:10.48550/arxiv.1912.07076

Cited by 75 publications

(113 citation statements)

References 0 publications

Supporting

Mentioning

113

Contrasting

Order By: Relevance

“…These models were found to perform well on tasks involving code-mixed text too [1,18]. Given that these models have to support over 100 languages with the limited model capacity they have, some works have found that they are outperformed by monolingual versions, even on some low-resourced languages [8,25,27,31]. Some of the aforementioned works [26,33,34] have theorized about the reasons why crosslingual transfer works and have stated that factors like pretraining data size and vocabulary overlap between languages could affect the transfer performance of a language in these models.…”

Section: Related Workmentioning

confidence: 99%

Predicting the Performance of Multilingual NLP Models

Srinivasan¹,

Sitaram²,

Ganu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent advancements in NLP have given us models like mBERT and XLMR that can serve over 100 languages. The languages that these models are evaluated on, however, are very few in number, and it is unlikely that evaluation datasets will cover all the languages that these models support. Potential solutions to the costly problem of dataset creation are to translate datasets to new languages or use template-filling based techniques for creation. This paper proposes an alternate solution for evaluating a model across languages which make use of the existing performance scores of the model on languages that a particular task has test sets for. We train a predictor on these performance scores and use this predictor to predict the model's performance in different evaluation settings. Our results show that our method is effective in filling the gaps in the evaluation for an existing set of languages, but might require additional improvements if we want it to generalize to unseen languages.

show abstract

Section: Related Workmentioning

confidence: 99%

Predicting the Performance of Multilingual NLP Models

Srinivasan¹,

Sitaram²,

Ganu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…BERT models have also become available as part of the multilingual BERT model 21 (Devlin et al, 2019) or trained separately for Finnish 22 23 (Kutuzov et al, 2017;Virtanen et al, 2019).…”

Section: Semanticsmentioning

confidence: 99%

The Current State of Finnish NLP

Hämäläinen¹,

Alnajjar²

2021

Preprint

View full text Add to dashboard Cite

There are a lot of tools and resources available for processing Finnish. In this paper, we survey recent papers focusing on Finnish NLP related to many different subcategories of NLP such as parsing, generation, semantics and speech. NLP research is conducted in many different research groups in Finland, and it is frequently the case that NLP tools and models resulting from academic research are made available for others to use on platforms such as Github. TiivistelmäSuomen kielen koneelliseen käsittelyyn on tarjolla paljon valmiita työkaluja ja resursseja.Tässä artikkelissa tarkastelemme viimeaikoina julkaistuja tieteellisiä artikkeleita, joissa keskitytään suomen kielen kieliteknologiaan.Tarkastelemme kieliteknologian eri alaluokkia, kuten jäsentämistä, tuottamista, semantiikkaa ja puheetta. kieliteknologista tutkimusta tehdään Suomessa monissa eri tutkimusryhmissä, ja usein akateemisen tutkimuksen tuloksena tuotetut kieliteknologian työkalut ja mallit julkaistaan muiden käytettäväksi esimerkiksi Githubissa.

show abstract

“…They are often based on BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al, 2019) and other contextualised architectures. A number of language-specific initiatives have in recent years released monolingual versions of these models for a number of languages (Fares et al, 2017;Kutuzov and Kuzmenko, 2017;Virtanen et al, 2019;de Vries et al, 2019;Ulčar and Robnik-Šikonja, 2020;Koutsikakis et al, 2020;Nguyen and Nguyen, 2020;Farahani et al, 2020;Malmsten et al, 2020). For our purposes, the most important such previous training effort is that of Virtanen et al (2019) on creating a BERT model for Finnish -FinBERT 5as our training setup for creating NorBERT builds heavily on this; see Section 6 for more details.…”

Section: Related Workmentioning

confidence: 99%

“…Our NorBERT model is trained from scratch for Norwegian, and can be used in exactly the same way as any other BERT-like model. The NorBERT training setup heavily builds on prior work on Fin-BERT conducted at the University of Turku (Virtanen et al, 2019).…”

Section: Norbertmentioning

confidence: 99%

Large-Scale Contextualised Language Modelling for Norwegian

Kutuzov,

Barnes,

Velldal

et al. 2021

Preprint

View full text Add to dashboard Cite

We present the ongoing NorLM initiative to support the creation and use of very large contextualised language models for Norwegian (and in principle other Nordic languages), including a ready-to-use software environment, as well as an experience report for data preparation and training. This paper introduces the first largescale monolingual language models for Norwegian, based on both the ELMo and BERT frameworks. In addition to detailing the training process, we present contrastive benchmark results on a suite of NLP tasks for Norwegian.

show abstract

Multilingual is not enough: BERT for Finnish

Cited by 75 publications

References 0 publications

Predicting the Performance of Multilingual NLP Models

Predicting the Performance of Multilingual NLP Models

The Current State of Finnish NLP

Large-Scale Contextualised Language Modelling for Norwegian

Contact Info

Product

Resources

About