Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop

Habash, Nizar; Rambow, Owen

doi:10.3115/1219840.1219911

Cited by 253 publications

(214 citation statements)

References 6 publications

Supporting

Mentioning

212

Contrasting

Order By: Relevance

“…MADAMIRA (Pasha et al, 2014); the current state-of-the-art tool for Arabic morphological analysis and disambiguation, obtains the disambiguated morphological analysis of the word, and feeds it to a tokenization engine. MADAMIRA utilizes MADA (Habash and Rambow, 2005;Roth et al, 2008) for morphological disambiguation. The top morphological analysis is then used for tokenization deterministically through one of the tokenization schemes.…”

Section: Background and Related Workmentioning

confidence: 99%

Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

Zalmout

Habash

2017

The Prague Bulletin of Mathematical Linguistics

Self Cite

View full text Add to dashboard Cite

Tokenization is very helpful for Statistical Machine Translation (SMT), especially when translating from morphologically rich languages. Typically, a single tokenization scheme is applied to the entire source-language text and regardless of the target language. In this paper, we evaluate the hypothesis that SMT performance may benefit from different tokenization schemes for different words within the same text, and also for different target languages. We apply this approach to Arabic as a source language, with five target languages of varying morphological complexity: English, French, Spanish, Russian and Chinese. Our results show that different target languages indeed require different source-language schemes; and a context-variable tokenization scheme can outperform a context-constant scheme with a statistically significant performance enhancement of about 1.4 BLEU points.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

Zalmout

Habash

2017

The Prague Bulletin of Mathematical Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the alignment task, the data was tokenized and lowercased for English, and transliterated and segmented using MADA [2] for Arabic. Table 2 shows the correspondences between the one of the seven English connective "while" and Arabic translations detected automatically using the annotation projection from English sentences to Arabic ones.…”

Section: Towards a Multilingual Act Metricmentioning

confidence: 99%

Assessing the Accuracy of Discourse Connective Translations: Validation of an Automatic Metric

Hajlaoui

Popescu-Belis

2013

Computational Linguistics and Intelligent Text Processing

View full text Add to dashboard Cite

Abstract. Automatic metrics for the evaluation of machine translation (MT) compute scores that characterize globally certain aspects of MT quality such as adequacy and fluency. This paper introduces a reference-based metric that is focused on a particular class of function words, namely discourse connectives, of particular importance for text structuring, and rather challenging for MT. To measure the accuracy of connective translation (ACT), the metric relies on automatic word-level alignment between a source sentence and respectively the reference and candidate translations, along with other heuristics for comparing translations of discourse connectives. Using a dictionary of equivalents, the translations are scored automatically, or, for better precision, semiautomatically. The precision of the ACT metric is assessed by human judges on sample data for English/French and English/Arabic translations: the ACT scores are on average within 2% of human scores. The ACT metric is then applied to several commercial and research MT systems, providing an assessment of their performance on discourse connectives.

show abstract

“…The raw Arabic text is enriched and tokenized using the Morphological Analysis and Disambiguation for Arabic (MADA) toolkit (Habash and Rambow, 2005). The various Arabic tokenization schemes that we experiment with, span a segmentation spectrum ranging from coarse segmentation, which uses unsegmented text, to fine segmentation which splits off all possible clitics.…”

Section: Arabic Preprocessing Schemesmentioning

confidence: 99%

“…enriched and the different tokenization generated using the Morphological Analysis and Disambiguation for Arabic (MADA) toolkit (Habash and Rambow, 2005). The parallel training corpora was then filtered by first removing sentence pairs longer than 99 on either side then deleting unbalanced sentence pairs with ratio more than a 4-to-1 in either direction.…”

Section: Training Datamentioning

confidence: 99%

The impact of Arabic morphological segmentation on broad-coverage English-to-Arabic statistical machine translation

Al-Haj

Lavie

2011

Machine Translation

View full text Add to dashboard Cite

Morphologically rich languages pose a challenge for statistical machine translation (SMT). This challenge is magnified when translating into a morphologically rich language. In this work we address this challenge in the framework of a broad-coverage English-to-Arabic phrase based statistical machine translation (PBSMT). We explore the full spectrum of Arabic segmentation schemes ranging from full word form to fully segmented forms and examine the effects on system performance. Our results show a difference of 2.61 BLEU points between the best and worst segmentation schemes indicating that the choice of the segmentation scheme has a significant effect on the performance of a PBSMT system in a large data scenario. We also show that a simple segmentation scheme can perform as good as the best and more complicated segmentation scheme. We also report results on a wide set of techniques for recombining the segmented Arabic output.

show abstract

Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop

Cited by 253 publications

References 6 publications

Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

Assessing the Accuracy of Discourse Connective Translations: Validation of an Automatic Metric

The impact of Arabic morphological segmentation on broad-coverage English-to-Arabic statistical machine translation

Contact Info

Product

Resources

About