A Multitask Learning Approach for Diacritic Restoration

Abstract

In many languages like Arabic, diacritics are used to specify pronunciations as well as meanings. Such diacritics are often omitted in written text, increasing the number of possible pronunciations and meanings for a word. This results in a more ambiguous text making computational processing on such text more difficult. Diacritic restoration is the task of restoring missing diacritics in the written text. Most state-of-the-art diacritic restoration models are built on character level information which helps generalize the model to unseen data, but presumably lose useful information at the word level. Thus, to compensate for this loss, we investigate the use of multi-task learning to jointly optimize diacritic restoration with related NLP problems namely word segmentation, part-of-speech tagging, and syntactic diacritization. We use Arabic as a case study since it has sufficient data resources for tasks that we consider in our joint modeling. Our joint models significantly outperform the baselines and are comparable to the state-ofthe-art models that are more complex relying on morphological analyzers and/or a lot more data (e.g. dialectal data).

Multi-Task Learning in Natural Language Processing: An Overview

“…The direction of hierarchical pipelines is not necessarily always from low-level tasks to high-level tasks. For example, in [1], the outputs of word-level tasks are fed to the char-level primary task. [99] feeds the output of more general classification models to more specific classification models during training, and the more general classification results are used to optimize beam search of more specific models at test time.…”

“…Besides, another common practice is to share the first embedding layers across tasks as [58,156] did. [1] shares word and character embedding matrices and combines them differently for different tasks. [103] shares two encoding layers and a vocabulary lookup table between the primary neural machine translation task and the Relevance-based Auxiliary Task (RAT).…”

Improving Arabic Diacritization with Regularized Decoding and Adversarial Training

“…It is worth noting that previous studies (Zitouni et al, 2006;Arabiyat, 2015;Fadel et al, 2019a;Abandah and Abdel-Karim, 2019;Alqahtani et al, 2019Alqahtani et al, , 2020 use different methods to compute diacritic error rate (DER) for ATB and Tashikeela datasets. Therefore, we follow the schema in Zitouni et al (2006); Arabiyat (2015); Abandah and Abdel-Karim (2019) to compute DER for ATB and follow Fadel et al (2019a); Alqahtani et al (2019Alqahtani et al ( , 2020 to compute that for Tashikeela.…”

Recent Advances in Arabic Syntactic Diacritics Restoration

“…Restoring Arabic syntactic diacritics based on Long Short-Term Memory (LSTM) networks leads to state-ofthe-art performance [3,30,31,32]. These word level LSTM networks are commonly augmented with Maximum Entropy (MaxEnt) sparse direct connections between the input and the output layers of the tagger [3].…”