RobeCzech: Czech RoBERTa, a Monolingual Contextualized Language Representation Model

Straka, Milan; Náplava, Jakub; Straková, Jana; Samuel, David

doi:10.1007/978-3-030-83527-9_17

Cited by 24 publications

(14 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The modifications included a larger training process, to remove the next sentence prediction objective, use of longer sequences and a dynamic use of the mask applied over the training data. As in the case of BERT, RoBERTa has been fine‐tuned for specific languages such as dutch (Delobelle et al, 2020) or czech (Straka et al, 2021) and for many specific problems such as metaphor identification (Babieno et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

BERTuit: Understanding Spanish language in Twitter with transformers

2023

View full text Add to dashboard Cite

The appearance of complex attention-based language models such as BERT, RoBERTa or GPT-3 has allowed to address highly complex tasks in a plethora of scenarios. However, when applied to specific domains, these models encounter considerable difficulties. This is the case of Social Networks such as Twitter, an everchanging stream of information written with informal and complex language, where each message requires careful evaluation to be understood even by humans given the important role that context plays. Addressing tasks in this domain through Natural Language Processing involves severe challenges. When powerful state-of-the-art multilingual language models are applied to this scenario, language specific nuances get lost in translation. To face these challenges we present BERTuit, the largest transformer proposed so far for Spanish language, pre-trained on a massive dataset of 230 M Spanish tweets using RoBERTa optimization. Our motivation is to provide a powerful resource to better understand Spanish Twitter and to be used on applications focused on this social network, with special emphasis on solutions devoted to tackle the spreading of misinformation in this platform. BERTuit is evaluated on several tasks and compared against M-BERT, XLM-RoBERTa and XLM-T, very competitive multilingual transformers. The utility of our approach is shown with applications, in this case: an unsupervised methodology to visualize groups of hoaxes; and supervised profiling of authors spreading disinformation.

show abstract

Section: Related Workmentioning

confidence: 99%

BERTuit: Understanding Spanish language in Twitter with transformers

2023

View full text Add to dashboard Cite

show abstract

“…They further propose that, with the only little tuning of hyperparameters, the model outperformed all other tested German and multilingual BERT models. A monolingual RoBERTa language model trained on Czech data has been presented in [21]. The authors show that the model significantly outperforms equally-sized multilingual and Czech language-oriented model variants.…”

Section: Related Workmentioning

confidence: 99%

Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi

Velankar¹,

Patil²,

Joshi³

2022

Preprint

View full text Add to dashboard Cite

Transformers are the most eminent architectures used for a vast range of Natural Language Processing tasks. These models are pre-trained over a large text corpus and are meant to serve state-of-the-art results over tasks like text classification. In this work, we conduct a comparative study between monolingual and multilingual BERT models. We focus on the Marathi language and evaluate the models on the datasets for hate speech detection, sentiment analysis, and simple text classification in Marathi. We use standard multilingual models such as mBERT, indicBERT, and xlm-RoBERTa and compare them with MahaBERT, MahaALBERT, and MahaRoBERTa, the monolingual models for Marathi. We further show that Marathi monolingual models outperform the multilingual BERT variants in five different downstream fine-tuning experiments. We also evaluate sentence embeddings from these models by freezing the BERT encoder layers. We show that monolingual MahaBERT-based models provide rich representations as compared to sentence embeddings from multi-lingual counterparts. However, we observe that these embeddings are not generic enough and do not work well on out-of-domain social media datasets. We consider two Marathi hate speech datasets L3Cube-MahaHate, HASOC-2021, a Marathi sentiment classification dataset L3Cube-MahaSent, and Marathi Headline, Articles classification datasets.

show abstract

“…Finally, we have performed a round of experiments with a pair of recently published Czech monolingual models. RobeCzech [48] was pretrained on a set of currated Czech corpora using the RoBERTa-base architecture. FERNET-C5 [49] was pretrained on a large crawled dataset, using Bert-base architecture.…”

Section: Natural Language Inferencementioning

confidence: 99%

CsFEVER and CTKFacts: Acquiring Czech data for fact verification

Drchal¹,

Ullrich²,

Rýpar³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we present two Czech datasets for automated factchecking, which is a task commonly modeled as a classification of textual claim veracity w.r.t. a corpus of trusted ground truths. We consider 3 classes: SUPPORTS, REFUTES complemented with evidence documents or NEI (Not Enough Info) alone. Our first dataset, CsFEVER, has 127,328 claims. It is an automatically generated Czech version of the large-scale FEVER dataset built on top of Wikipedia corpus. We take a hybrid approach of machine translation and document alignment; the approach, and the tools we provide, can be easily applied to other languages. The second dataset, CTKFacts of 3,097 claims, is annotated using the corpus of 2.2M articles of Czech News Agency. We present its extended annotation methodology based on the FEVER approach. We analyze both datasets for spurious cues -annotation patterns leading to model overfitting. CTKFacts is further examined for inter-annotator agreement, thoroughly cleaned,

show abstract

RobeCzech: Czech RoBERTa, a Monolingual Contextualized Language Representation Model

Cited by 24 publications

References 24 publications

BERTuit: Understanding Spanish language in Twitter with transformers

BERTuit: Understanding Spanish language in Twitter with transformers

Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi

CsFEVER and CTKFacts: Acquiring Czech data for fact verification

Contact Info

Product

Resources

About