GREEK-BERT: The Greeks visiting Sesame Street

Abstract

Transformer-based language models, such as BERT and its variants, have achieved state-of-the-art performance in several downstream natural language processing (NLP) tasks on generic benchmark datasets (e.g., GLUE, SQUAD, RACE). However, these models have mostly been applied to the resource-rich English language. In this paper, we present GREEK-BERT, a monolingual BERT-based language model for modern Greek. We evaluate its performance in three NLP tasks, i.e., part-of-speech tagging, named entity recognition, and natural language inference, obtaining state-of-the-art performance. Interestingly, in two of the benchmarks GREEK-BERT outperforms two multilingual Transformer-based models (M-BERT, XLM-R), as well as shallower neural baselines operating on pretrained word embeddings, by a large margin (5%-10%). Most importantly, we make both GREEK-BERT and our training code publicly available, along with code illustrating how GREEK-BERT can be fine-tuned for downstream NLP tasks. We expect these resources to boost NLP research and applications for modern Greek.

Scalar Adjective Identification and Multilingual Ranking

“…We use the pre-trained cased and uncased multilingual BERT model (Devlin et al, 2019) and report results of the best variant for each language. We also report results obtained with four monolingual models: bert-base-uncased (Devlin et al, 2019), flaubert_base_uncased (Le et al, 2020), bert-base-spanish-wwmuncased (Cañete et al, 2020), and bert-basegreek-uncased-v1 (Koutsikakis et al, 2020). We compare to results obtained using fastText static embeddings in each language (Grave et al, 2018).…”

A Survey on Sentiment Analysis and Opinion Mining in Greek Social Media

“…Very recently, researchers employed transformer-based language models, such as BERT, and achieved state-of-the-art performance in several NLP tasks. The GreekBERT model [54] has been pre-trained on 29 GB of text from the Greek part of (i) Wikipedia, (ii) the European Parliament Proceedings Parallel Corpus (Europarl), and (iii) of OSCAR [55], a clean version of Common Crawl [56]. Finally, a GPT-2 model for Greek trained on a large text corpus (almost 5GB) from Wikipedia, is also available online [57].…”

“…In addition to PaloBERT, we decided to further train an existing language model on the collected dataset. More specifically, we selected GreekBERT [54] (Section 2.3) and we trained it for an additional 10 epochs (Figure 6) on our social media corpus, on an AWS p2.xlarge instance running the Deep Learning AMI (Amazon Linux) Version 48.0. The resulting model, which is further mentioned as GreekSocialBERT, is also available for download at HuggingFace's model repository [73].…”

Let’s PlayMono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

“…We use the context representations from a 600-d context2vec model pre-trained on the ukWaC corpus (Baroni et al, 2009). 11 For French, Spanish, and Greek, we use BERT models specifically trained for each language: Flaubert (flaubert base uncased) , BETO (bert-base-spanish-wwmuncased) (Cañete et al, 2020), and Greek BERT (bert-base-greek-uncased-v1) (Koutsikakis et al, 2020). We also use the bertbase-multilingual-cased model for each of the four languages.…”