ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

Abstract

Pre-trained language models (LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-theart results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large (∼ 3.4× larger size). Our models are publicly available at https://github.com/UBC-NLP/marbert and ARLUE will be released through the same repository.

Emotion Analysis of Arabic Tweets: Language Models and Available Resources

“…Only two studies, however, attempted to use PLMs and TL for the task of Arabic emotion classification. The first model was a BERT-based model named MARBERT (Abdul-Mageed et al, 2020 ), which was trained on tweets, and social meaning tasks were considered in the fine-tuning. The social meaning tasks included emotion detection.…”

“…Textual data on Twitter are frequently dialectal, with dialects lacking spelling norms and informal in nature, and may include emojis, hashtags, and user mentions. Only QARiB (Abdelali et al, 2021 ) and MARBERT (Abdul-Mageed et al, 2020 ), as mentioned above, used tweets to train transformer models and fine-tune for emotion classification tasks, and both of them are BERT-based. Moreover, even when testing on informal text, increasing the variety of training data by including both formal (MSA) and informal (AD) text is better than using informal text alone (Abdelali et al, 2021 ).…”

Detecting Arabic textual threats in social media using artificial intelligence: An overview

“…Transformers have recently become popular in NLP and text categorization. The transformer model is a deep neural network architecture based completely on the attention mechanism, replacing the recurrent layers with auto encoder-decoder architectures with special called multi-head self-attention layers [41].…”

Arabic Offensive and Hate Speech Detection Using a Cross-Corpora Multi-Task Learning Model

“…El Mahdaouy et al [64] approached the same shared task using an end-to-end multitask learning model based on the MarBERT [65] language model. Duwairi et al [66] investigated the ability of CNN, CNN-LSTM, and BiLSTM-CNN networks to detect hateful content on social media.…”

“…Each word in the inputted sequence will be mapped to an embedding vector generated from summing up its corresponding word, segment, and positional embeddings. To learn shared global contextual representations across all corpora, the shared part is employed to fine-tune the weights of a pre-trained multilayer bidirectional transformer encoder such as AraBERT [41], and MarBERT [65]. The AraBERT and MarBERT are transformer encoders consisting of a self-attention mechanism to learn the contextual representations for each inputted word.…”