BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets

Guo, Yanzhu; Rennard, Virgile; Xypolopoulos, Christos; Vazirgiannis, Michalis

doi:10.48550/arxiv.2109.10234

Cited by 4 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Continual learning aims to specialize a PLM to a particular domain by continuing the pretraining task on the abundant unlabeled corpora, such as biological documents (BioBERT [90]) scientific papers (SciBERT [91]), clinical notes (ClinicalBERT [92], [93], ClinicalCLNet [94]), financial new (FinBERT [11]), legal documents (LegalBERT [12]) and tweets (BERTweet [95] for English tweets and BERTweetFR [96] for french tweets). During this stage, we continue to train the PLM using the same pretraining task, which is usually a language modeling objective, on the target domain datasets.…”

Section: A Continual Learningmentioning

confidence: 99%

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

Xu¹,

Han²

2022

Preprint

View full text Add to dashboard Cite

Recent advances in NLP are brought by a range of large-scale pretrained language models (PLMs). These PLMs have brought significant performance gains for a range of NLP tasks, circumventing the need to customize complex designs for specific tasks. However, most current work focus on finetuning PLMs on a domain-specific datasets, ignoring the fact that the domain gap can lead to overfitting and even performance drop. Therefore, it is practically important to find an appropriate method to effectively adapt PLMs to a target domain of interest. Recently, a range of methods have been proposed to achieve this purpose. Early surveys on domain adaptation are not suitable for PLMs due to the sophisticated behavior exhibited by PLMs from traditional models trained from scratch and that domain adaptation of PLMs need to be redesigned to take effect. This paper aims to provide a survey on these newly proposed methods and shed light in how to apply traditional machine learning methods to newly evolved and future technologies. By examining the issues of deploying PLMs for downstream tasks, we propose a taxonomy of domain adaptation approaches from a machine learning system view, covering methods for input augmentation, model optimization and personalization. We discuss and compare those methods and suggest promising future research directions.

show abstract

Section: A Continual Learningmentioning

confidence: 99%

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

Xu¹,

Han²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Given the difference in writing formality on Twitter, we use Twitter-based in addition to general-purpose models. For the former, XLM-T [2] is our multilingual model, BERTweet [20] for English, RoBERTuito [23] for Spanish, AraBERT [1] for Arabic, and BERTweetFR [14] for French. For the latter, we use multilingual BERT [9] as our multilingual model, BERT for English and Arabic, and BETO [6] for Spanish.…”

Section: Implementation Detailsmentioning

confidence: 99%

Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats

Chochlakis¹,

Mahajan²,

Baruah³

et al. 2022

Preprint

View full text Add to dashboard Cite

The need for emotional inference from text continues to diversify as more and more disciplines integrate emotions into their theories and applications. These needs include inferring different emotion types, handling multiple languages, and different annotation formats. A shared model between different configurations would enable the sharing of knowledge and a decrease in training costs, and would simplify the process of deploying emotion recognition models in novel environments. In this work, we study how we can build a single model that can transition between these different configurations by leveraging multilingual models and Demux, a transformer-based model whose input includes the emotions of interest, enabling us to dynamically change the emotions predicted by the model. Demux also produces emotion embeddings, and performing operations on them allows us to transition to clusters of emotions by pooling the embeddings of each cluster. We show that Demux can simultaneously transfer knowledge in a zero-shot manner to a new language, to a novel annotation format and to unseen emotions. Code is available at https://github.com/gchochla/Demux-MEmo.

show abstract

“…We also introduce BERTweetFR [Guo+21], the first large-scale pre-trained language model for French tweets. As a valuable resource for social media data, tweets are often written in an informal tone and have their own set of characteristics compared to conventional sources.…”

Section: Large Scale Linguistic Resourcesmentioning

confidence: 99%

NLP Research and Resources at DaSciM, Ecole Polytechnique

Abdine¹,

Guo²,

Eddine³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

DaSciM (Data Science and Mining) part of LIX at Ecole Polytechnique, established in 2013 and since then producing research results in the area of large scale data analysis via methods of machine and deep learning. The group has been specifically active in the area of NLP and text mining with interesting results at methodological and resources level. Here follow our different contributions of interest to the AFIA community.

show abstract

BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets

Cited by 4 publications

References 15 publications

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats

NLP Research and Resources at DaSciM, Ecole Polytechnique

Contact Info

Product

Resources

About