An open access NLP dataset for Arabic dialects : Data collection, labeling, and model construction

Abstract

Natural Language Processing (NLP) is today a very active field of research and innovation. Many applications need however big sets of data for supervised learning, suitably labelled for the training purpose. This includes applications for the Arabic language and its national dialects. However, such open access labeled data sets in Arabic and its dialects are lacking in the Data Science ecosystem and this lack can be a burden to innovation and research in this field. In this work, we present an open data set of social data content in several Arabic dialects. This data was collected from the Twitter social network and consists on +50K twits in five (5) national dialects. Furthermore, this data was labeled for several applications, namely dialect detection, topic detection and sentiment analysis. We publish this data as an open access data to encourage innovation and encourage other works in the field of NLP for Arabic dialects and social media. A selection of models were built using this data set and are presented in this paper along with their performances.

Enhancing Arabic Dialect Detection on Social Media: A Hybrid Model with an Attention Mechanism

“…Ali et al [11] developed a way to differentiate between Arabic dialects distinctly by taking bottleneck characteristics from the i-vector framework and phonetic and lexical data from a speech recognition system, which are then used to identify dialects in Arabic broadcast speech. Similarly, Boujou et al [12] introduced an open data collection of social data material in Arabic dialects obtained from the Twitter social network. The researchers then evaluated the dataset using four different classifiers, namely SGD, LR, NB, and linear SVC, with NB achieving the highest accuracy of 0.79%.…”

“…Many attempts have been proposed in the area of automatic dialect identification (ADI), and early uses are based on dictionaries, rules, and language modeling [5][6][7][8][9][10]; more recently, a shift was made toward employing machine learning techniques [11][12][13][14][15][16][17][18][19][20][21][22][23][24], deep learning approaches [25][26][27][28][29][30][31][32][33][34][35][36][37][38][39][40], and transfer learning methods [41][42][43][44][45][46][47][48][49]. Many of these investigations utilize prominent and accessible datasets, such as MADAR [49], NADI [50][51]…”

Improvement of Moroccan Dialect Sentiment Analysis Using Arabic BERT-Based Models

“…Our observation revealed a scarcity of open-source datasets available for MD. Consequently, in our research, we opted to work with two selected datasets to address this limitation: The Modelling Simulation and Data Analysis (MSDA) dataset (Boujou et al, 2021) and the Moroccan Arabic Corpus (MAC) dataset (Garouani and Kharroubi, 2021). We also combined the two to obtain a balanced dataset, the MSDA-MAC dataset:…”

Twitter Sentiment Analysis for Reviewing Tourist Destinations in Saudi Arabia using Apache Spark and Machine Learning Algorithms

“…Noise removal is the process of removing characters, numbers and pieces of text that may interfere with the analysis (Boujou et al, 2021). Noise removal is one of the most required steps in text pre-processing.…”

“…Another important step of the text pre-processing is removing the stop-words from the text (Boujou et al, 2021). Stop-words appear too frequently in any type of text.…”