Tag based models for Arabic text compression

Classifying and Segmenting Classical and Modern Standard Arabic using Minimum Cross-Entropy

“…It produces state-of-theart text compression results for many languages as detailed in the reports mentioned in [31], [36], [57]. PPM has been used as the basis for an effective method for performing Chinese word segmentation where spaces (as word separators) are inserted into Chinese text which has no spaces [33].…”

“…PPM has been used as the basis for an effective method for performing Chinese word segmentation where spaces (as word separators) are inserted into Chinese text which has no spaces [33]. Other studies such as [31], [34]- [36], [57], [58] have reported using PPM for different languages for other NLP tasks such as cryptology, code switching, authorship attribution, text correction and speech recognition.…”

“…Since most popular recognised Arabic POS taggers were trained on MSA text [36], [49], the tagging of mixed corpora text will vary in quality and will not be consistent and reliable. The final experiment was conducted using tag-based compression on mixed Arabic corpora, selected using the previous outcomes in order to evaluate the process of both the tagging and classification.…”

“…This Markov-based approach effectively uses character-based language models and has been employed in many NLP tasks in the past often with state-of-the-art results or results competitive with traditional schemes [1], [31]- [36].…”

“…One example of those tasks is parts-of-speech tagging of Arabic language. Most of the popular Arabic partsof-speech or POS taggers were trained on MSA text [36], [49], and therefore, the performance of the taggers will be best when tagging MSA text [36]. Contrastingly, tagging Classical Arabic text using MSA POS taggers will significantly reduce the quality of the tagging [47].…”