CamemBERT: a Tasty French Language Model

Abstract

Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models-in all languages except English-very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

On the Use of Self-Supervised Pre-Trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition

“…In this study, linguistic representation will be used to process French data. Very recently, CamemBERT [26], FlauBERT [27] and GermanBERT 1 models were released for French and German while Ernie models are only available in Chinese and English. As far as the authors know, this is the first time that such models are used for feature extraction in order to perform SER tasks.…”

De-identification of Emergency Medical Records in French: Survey and Comparison of State-of-the-Art Automated Systems

“…A benchmark of NER models on French commercial legal cases has been developed (Benesty 2019). The results encourage the use of the NER bi-directional long short term memory (Bi-LSTM) model by using the Flair library (Akbik, Blythe, and Vollgraf 2018) and the NER model of CamemBERT (Martin et al 2020). CamemBERT is a French version of BERT -Bidirectional Encoder Representations from Transformers- (Devlin et al 2018), which is itself based on the encoder part of the Transformer architecture (Vaswani et al 2017).…”

Overview of the Transformer-based Models for NLP Tasks

“…In order to tackle specific languages problems, different monolingual versions of BERT were trained in different languages. For example BERTje [36] is a Dutch version, AlBERTo [37] is an Italian version, and CamemBERT [38] and FlauBERT [39] are two different models for French. These models outperform vanilla BERT in different NLP tasks specific to these languages.…”