BERTić -- The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbian

Abstract

In this paper we describe a transformer model pre-trained on 8 billion tokens of crawled text from the Croatian, Bosnian, Serbian and Montenegrin web domains. We evaluate the transformer model on the tasks of partof-speech tagging, named-entity-recognition, geo-location prediction and commonsense causal reasoning, showing improvements on all tasks over state-of-the-art models. For commonsense reasoning evaluation we introduce COPA-HR -a translation of the Choice of Plausible Alternatives (COPA) dataset into Croatian. The BERTić model is made available for free usage and further task-specific fine-tuning through HuggingFace.

Too Many Cooks Spoil the Model: Are Bilingual Models for Slovene Better than a Large Multilingual Model?

“…This same motivation has also led to a transformer model that exclusively uses languages of the Slavic genus. The BERTić language model (Ljubešić and Lauc, 2021) was trained from scratch for Bosnian, Croatian, Montenegrin and Serbian. Whereas the CroSloEngual model uses more distant languages, BERTić selected these languages because they are very closely related, are mutually intelligible and because they are considered part of the same Serbo-Croatian macro language (according to the ISO 639-3 Macrolanguage Mappings).…”

Creating a stop word dictionary in Serbian

“…The first, and, at the time of writing of this paper, the only transformerbased language model specifically trained for Serbian, Croatian, Bosnian, and Montenegrin is BERTić (Ljubešić and Lauc, 2021). BERTić is trained using the ELECTRA approach (Clark et al, 2020) for training transformer models.…”

“…The BERTić-data text collection contains more than 8 billion tokens of mostly web-crawled text written in Bosnian, Croatian, Montenegrin, or Serbian. The collection was used to train the BERTić transformer model (Ljubešić and Lauc, 2021). The Wikipedia dumps of the Bosnian, Croatian, Macedonian, Montenegrin, Serbian, Serbo-Croatian, and Slovenian Wikipedias were collected in the comparable corpus CLASSLA-Wikipedia (CLASSLA-Wiki, Table 2).…”

“…A sentiment analysis approach was applied at the sentence level. The results of classification four of the transformer models were compared: FastText (Bojanowski et al, 2017) with pre-trained CLARIN.SI word embeddings (Ljubešić and Erjavec, 2018), XLM-R , CroSloEngual BERT (Ulčar and Robnik-Šikonja, 2020), and BERTić (Ljubešić and Lauc, 2021). BERTić gave the best results compared to the others (model macro F1 0.7941 ± 0.0101).…”

“…The transformer-based model was also introduced for several tasks in Serbian, Croatian, and Slovene, including NER (Ljubešić and Lauc, 2021). The model was pre-trained on web-crawled texts in Serbian, Bosnian, Croatian, and Slovene, consisting of 8 billion tokens, and then fine-tuned for NER on several openly available datasets, such as SETimes.SR , corpora of news articles, or ReLDI-sr (Ljubešić et al, 2017), corpora of annotated tweets.…”