Collaboratively Constructed Linguistic Resources for Language Variants and their Exploitation in NLP Application – the case of Tunisian Arabic and the Social Media

Abstract

Modern Standard Arabic (MSA) is the formal language in most Arabic countries. Arabic Dialects (AD) or daily language differs from MSA especially in social media communication. However, most Arabic social media texts have mixed forms and many variations especially between MSA and AD. This paper aims to bridge the gap between MSA and AD by providing a framework for the translation of texts of social media. More precisely, this paper focuses on the Tunisian Dialect of Arabic (TAD) with an application on automatic machine translation for a social media text into MSA and any other target language. Linguistic tools such as a bilingual TAD-MSA lexicon and a set of grammatical mapping rules are collaboratively constructed and exploited in addition to a language model to produce MSA sentences of Tunisian dialectal sentences. This work is a first-step towards collaboratively constructed semantic and lexical resources for Arabic Social Media within the ASMAT (Arabic Social Media Analysis Tools) project.

Machine translation for Arabic dialects (survey)

“…(deep morphological corpus of 1500 sentence pairs Tunisian-to-MSA 84% representation of data) Dev/test set 750 sentence pairs, MSA-to-Tunisian 80% (Sadat et al, 2014) Rule-based approach 50 sentences BLEU score: 14.32 +Bilingual lexicon+LM (Tachicart and Bouzoubaa, 2014) Rule-based approach -+Bilingual lexicon+LM (Meftouh et al, 2015) Statistical approach 6 sides parallel corpus A set of BLEU scores of 6400 sentences Dev/test set 500 sentence for each corpus…”

“…The best results of translation were achieved between the dialects of Algeria which is not a surprising result since they share a large part of the vocabulary. It was also shown that the (Bakr et al, 2008) Egyptian MSA (Salloum and Habash, 2012) Levantine, Egyptian, MSA Iraqi, Gulf Arabic (Mohamed et al, 2012) MSA Egyptian (Al-Gaphari and Al-Yadoumi, 2012) Sanaani (Yemenite) MSA (Hamdi et al, 2013) Tunisian MSA MSA Tunisian (Tachicart and Bouzoubaa, 2014) Moroccan MSA (Sadat et al, 2014) Tunisian MSA (Meftouh et al, 2015) Algerian, Tunisian, MSA Syrian and Palestinian MSA Algerian, Tunisian, Syrian and Palestinian performance of machine translation between Palestinian and Syrian was relatively high because of the closeness of the two dialects. Concerning MSA, the best results of machine translation have been achieved with Palestinian dialect.…”

“…In the generation step, MSA phrases are produced then passed to a language model to produce the most fluent MSA sentences (no evaluation was given for this work). Sadat et al (2014) provided a framework for translating Tunisian dialect text of social media into MSA. The work is based on a bilingual lexicon created for this context.…”

“…, (Sawaf, 2010), (Salloum and Habash, 2012), (Zbib et al, 2012), (Mohamed et al, 2012) (Salloum and Habash, 2013), (Sajjad et al, 2013), (Jeblee et al, 2014), (Aminian et al, 2014), (Durrani et al, 2014), (Salloum et al, 2014), (May et al, 2014), (Aransa, 2015), (Van der Wees et al, 2016) Levantine (Salloum and Habash, 2012), (Sawaf, 2010), (Meftouh et al, 2015) (Zbib et al, 2012), (Salloum and Habash, 2013), (Salloum et al, 2014) Tunisian (Hamdi et al, 2013), (Sawaf, 2010) (Sadat et al, 2014), (Meftouh et al, 2015) Iraqi (Salloum and Habash, 2012) (Sawaf, 2010), (Salloum and Habash, 2013) Gulf Arabic (Salloum and Habash, 2012) (Sawaf, 2010), (Salloum and Habash, 2013 • In terms of methodology, for translating between MSA and dialects the rule based approach with morphological analysis is the most used method. In addition, most work exploit bilingual lexicons and rely on relatively small language models (see data description in Table 4) compared to those used for standard languages.…”

A Multitask-Based Neural Machine Translation Model with Part-of-Speech Tags Integration for Arabic Dialects

“…The described system works with Levantine (Jordanian, Syrian and Palestinian), Egyptian, Iraqi and Gulf Arabic dialects. Sadat [9] presented a model for the translation of the Tunisian Arabic Dialect to the standardized modern form of Arabic. The model is based on a bilingual lexicon which was designed for the particular context of the translation exercise and uses a set of grammatical mapping rules with an additional step for disambiguation.…”