Handling OOV Words in Dialectal Arabic to English Machine Translation

Abstract

Dialects and standard forms of a language typically share a set of cognates that could bear the same meaning in both varieties or only be shared homographs but serve as faux amis. Moreover, there are words that are used exclusively in the dialect or the standard variety. Both phenomena, faux amis and exclusive vocabulary, are considered out of vocabulary (OOV) phenomena. In this paper, we present this problem of OOV in the context of machine translation. We present a new approach for dialect to English Statistical Machine Translation (SMT) enhancement based on normalizing dialectal language into standard form to provide equivalents to address both aspects of the OOV problem posited by dialectal language use. We specifically focus on Arabic to English SMT. We use two publicly available dialect identification tools: AIDA and MADAMIRA, to identify and replace dialectal Arabic OOV words with their modern standard Arabic (MSA) equivalents. The results of evaluation on two blind test sets show that using AIDA to identify and replace MSA equivalents enhances translation results by 0.4% absolute BLEU (1.6% relative BLEU) and using MADAMIRA achieves 0.3% absolute BLEU (1.2% relative BLEU) enhancement over the baseline. We show our replacement scheme reaches a noticeable enhancement in SMT performance for faux amis words.

Machine translation for Arabic dialects (survey)

“…The availability of some parallel corpora makes this research direction possible. Furthermore, availability of new tools related to dialect identification (at word and sentences levels) has a positive impact on machine trans-lation performance as it was shown in (Aminian et al, 2014;Salloum et al, 2014). Indeed, in this last work, identifying either the sentence is dialectal or MSA guides the selection of the MT system to use.…”

“…The performance of the system improves by 2.3 BLEU points when pivoting through MSA for first experiment, but when adding more dialectal data to training set (400k words) direct translation becomes better than mapping to MSA despite the significantly low OOV rate with MSA-mapping. Aminian et al (2014) dealt with OOV words in the context of Arabic to English SMT system. They adopted an approach that normalizes dialectal words to MSA words by using AIDA 6 (Elfardy et al, 2014) and MADAMIRA 7 (Pasha et al, 2014), to identify and replace dialectal Arabic OOV words with their MSA equivalents.…”

“…10 The task consisted in translating from Egyptian dialect and Mandarin Chinese into English. 11 It is worth noting that Egyptian dialect data within this campaign is a mixture of texts 8 Including letters and numbers 9 Open Machine Translation 2015 10 https://www.nist.gov/sites/default/files/documents/itl/iad/mig/ OpenMT15_EvalPlan_v0-9.pdf (Sawaf, 2010) Levantine, Gulf Arabic, Yes English Egyptian, Sudanese, Libyan, Moroccan, Tunisian (Salloum and Habash, 2011) Levantine, Egyptian Yes English (Zbib et al, 2012) Levantine, Egyptian No English (Salloum and Habash, 2013) Levantine, Egyptian, Yes English Iraqi, Gulf Arabic (Sajjad et al, 2013) Egyptian Yes English (Jeblee et al, 2014) English Yes Egyptian (Al-Mannai et al, 2014) Qatari No English (Durrani et al, 2014) Egyptian Yes English (Aminian et al, 2014) Egyptian Yes English (Salloum et al, 2014) Levantine, Egyptian Yes English (May et al, 2014) Egyptian No English (Aransa, 2015) Egyptian No English (Van der Wees et al, 2016) Egyptian No English in both Arabic script and Arabizi. In this respect, May et al (2014) presented a SMT system that translates informal Egyptian dialect to English which deals with Arabizi.…”

“…, (Sawaf, 2010), (Salloum and Habash, 2012), (Zbib et al, 2012), (Mohamed et al, 2012) (Salloum and Habash, 2013), (Sajjad et al, 2013), (Jeblee et al, 2014), (Aminian et al, 2014), (Durrani et al, 2014), (Salloum et al, 2014), (May et al, 2014), (Aransa, 2015), (Van der Wees et al, 2016) Levantine (Salloum and Habash, 2012), (Sawaf, 2010), (Meftouh et al, 2015) (Zbib et al, 2012), (Salloum and Habash, 2013), (Salloum et al, 2014) Tunisian (Hamdi et al, 2013), (Sawaf, 2010) (Sadat et al, 2014), (Meftouh et al, 2015) Iraqi (Salloum and Habash, 2012) (Sawaf, 2010), (Salloum and Habash, 2013) Gulf Arabic (Salloum and Habash, 2012) (Sawaf, 2010), (Salloum and Habash, 2013 • In terms of methodology, for translating between MSA and dialects the rule based approach with morphological analysis is the most used method. In addition, most work exploit bilingual lexicons and rely on relatively small language models (see data description in Table 4) compared to those used for standard languages.…”

Unsupervised False Friend Disambiguation Using Contextual Word Clusters and Parallel Word Alignments

“…The main difference between this approach and our previous work as described in (Aminian et al, 2014) lies in the fact that we try to improve SMT lexical choice by enhancing FF translation. Rather than blindly replacing all dialectal words with their standard equivalent as we did in (Aminian et al, 2014), here we try to automatically identify FF as one of the important sources of translation degradation across language variants and leverage knowledge acquired from monolingual standard data to predict the best equivalent for FF based on the context.…”