A Neural Architecture for Dialectal Arabic Segmentation

Samih, Younes; Attia, Mohammed; Eldesouki, Mohamed I.; Abdelali, Ahmed; Mubarak, Hamdy; Kallmeyer, Laura; Darwish, Kareem

doi:10.18653/v1/w17-1306

Cited by 53 publications

(17 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has been noted that stemming using Farasa improved the supervised sentiment classification performance in TEC, TAC datasets ( Table 2, Table 4) where it achieved the second best F1-score (85.9%) in TAC outperforming the baseline by 18.6%. Although Farasa was trained with MSA corpora, it succeeded in identifying the affixes to be cut in Tunisian words because of the lexical overlap between MSA and Arabic dialects in general [17]. In order to retain the variety of words having same root and different meanings, we have also used light stemming.…”

Section: Resultsmentioning

confidence: 99%

Tunisian Dialect Sentiment Analysis: A Natural Language Processing-based Approach

Mulki¹,

Haddad²,

Ali³

et al. 2018

CyS

View full text Add to dashboard Cite

Social media platforms have been witnessing a significant increase in posts written in the Tunisian dialect since the uprising in Tunisia at the end of 2010. Most of the posted tweets or comments reflect the impressions of the Tunisian public towards social, economical and political major events. These opinions have been tracked, analyzed and evaluated through sentiment analysis systems. In the current study, we investigate the impact of several preprocessing techniques on sentiment analysis using two sentiment classification models: Supervised and lexicon-based. These models were trained on three Tunisian datasets of different sizes and multiple domains. Our results emphasize the positive impact of preprocessing phase on the evaluation measures of both sentiment classifiers as the baseline was significantly outperformed when stemming, emoji recognition and negation detection tasks were applied. Moreover, integrating named entities with these tasks enhanced the lexicon-based classification performance in all datasets and that of the supervised model in medium and small sized datasets.

show abstract

Section: Resultsmentioning

confidence: 99%

Tunisian Dialect Sentiment Analysis: A Natural Language Processing-based Approach

Mulki¹,

Haddad²,

Ali³

et al. 2018

CyS

View full text Add to dashboard Cite

show abstract

“…It does not generalize well to other domains [8]. It has been shown by [12] that sequence to sequence mapping models work well in segmenting dialectical Arabic. In this work, we use a modified version of this second method.…”

Section: Segmentation Methodsmentioning

confidence: 94%

Arabic-SOS

Mohamed

Sayyed

2019

Proceedings of the 3rd International Conference on Digital Access to Textual Cultural Heritage

View full text Add to dashboard Cite

While morphological segmentation has always been a hot topic in Arabic, due to the morphological complexity of the language and the orthography, most effort has focused on Modern Standard Arabic. In this paper, we focus on pre-MSA texts. We use the Gradient Boosting algorithm to train a morphological segmenter with a corpus derived from Al-Manar, a late 19th/early 20th century magazine that focused on the Arabic and Islamic heritage. Since most of the cultural heritage Arabic available suffers from substandard orthography, we have trained a machine learner to standardize the text. Our segmentation accuracy reaches 98.47%, and the orthography standardization an F-macro of 0.98 and an F-micro of 0.99. We also produce stemming as a by-product of segmentation.

show abstract

“…The corpus contains about 200,000 words with POS categories, lemmatisation, English glosses and dialect identification annotations. Samih et al (2017) used a corpus, containing 350 tweets with more than 8000 words including 3000 words written in the Egyptian dialect. They manually annotated each word to provide: stem, lemma, CODA-compliant writing (Habash et al 2012a), segmentation and POS tags, as well as the corresponding MSA word, MSA segmentation and MSA POS tags.…”

Section: Morphological Analysis and Pos Taggingmentioning

confidence: 99%

Creation of annotated country-level dialectal Arabic resources: An unsupervised approach

Althobaiti

2021

Nat. Lang. Eng.

View full text Add to dashboard Cite

The wide usage of multiple spoken Arabic dialects on social networking sites stimulates increasing interest in Natural Language Processing (NLP) for dialectal Arabic (DA). Arabic dialects represent true linguistic diversity and differ from modern standard Arabic (MSA). In fact, the complexity and variety of these dialects make it insufficient to build one NLP system that is suitable for all of them. In comparison with MSA, the available datasets for various dialects are generally limited in terms of size, genre and scope. In this article, we present a novel approach that automatically develops an annotated country-level dialectal Arabic corpus and builds lists of words that encompass 15 Arabic dialects. The algorithm uses an iterative procedure consisting of two main components: automatic creation of lists for dialectal words and automatic creation of annotated Arabic dialect identification corpus. To our knowledge, our study is the first of its kind to examine and analyse the poor performance of the MSA part-of-speech tagger on dialectal Arabic contents and to exploit that in order to extract the dialectal words. The pointwise mutual information association measure and the geographical frequency of word occurrence online are used to classify dialectal words. The annotated dialectal Arabic corpus (Twt15DA), built using our algorithm, is collected from Twitter and consists of 311,785 tweets containing 3,858,459 words in total. We randomly selected a sample of 75 tweets per country, 1125 tweets in total, and conducted a manual dialect identification task by native speakers. The results show an average inter-annotator agreement score equal to 64%, which reflects satisfactory agreement considering the overlapping features of the 15 Arabic dialects.

show abstract

A Neural Architecture for Dialectal Arabic Segmentation

Cited by 53 publications

References 19 publications

Tunisian Dialect Sentiment Analysis: A Natural Language Processing-based Approach

Tunisian Dialect Sentiment Analysis: A Natural Language Processing-based Approach

Arabic-SOS

Creation of annotated country-level dialectal Arabic resources: An unsupervised approach

Contact Info

Product

Resources

About