SlovakBERT: Slovak Masked Language Model

Abstract

We introduce a new Slovak masked language model called SlovakBERT in this paper. It is the first Slovak-only transformers-based model trained on a sizeable corpus. We evaluate the model on several NLP tasks and achieve state-of-the-art results. We publish the masked language model, as well as the subsequently fine-tuned models for part-of-speech tagging, sentiment analysis and semantic textual similarity.

Review of Recent Trends in the Detection of Hate Speech and Offensive Language on Social Media

“…Another interesting study is from the Slovak researchers M. Pikuliak et al [14], who created a language model called SlovakBERT. This model has RoBERTa architecture (Liu et al [15]) and it is trained on a web-crawled corpus.…”