Arabic preprocessing schemes for statistical machine translation

Habash, Nizar; Sadat, Fatiha

doi:10.3115/1614049.1614062

Cited by 115 publications

(104 citation statements)

References 11 publications

Supporting

Mentioning

102

Contrasting

Order By: Relevance

“…Her results showed that morphological preprocessing helps, but only for smaller corpora. Habash and Sadat (2006) and Sadat and Habash (2006) reached similar conclusions on a much larger set of experiments including multiple preprocessing schemes reflecting different levels of morphological representation and multiple techniques for disambiguation and tokenization. They showed that specific preprocessing decisions can have a positive effect when decoding text with a different genre than that of the training data (in essence another form of data sparsity).…”

Section: Morphology-based Approachessupporting

confidence: 53%

Symbolic-to-statistical hybridization: extending generation-heavy machine translation

Habash

Dorr

Monz

2009

Machine Translation

Self Cite

View full text Add to dashboard Cite

The last few years have witnessed an increasing interest in hybridizing surface-based statistical approaches and rule-based symbolic approaches to machine translation (MT). Much of that work is focused on extending statistical MT systems with symbolic knowledge and components. In the brand of hybridization discussed here, we go in the opposite direction: adding statistical bilingual components to a symbolic system. Our base system is Generation-heavy machine translation (GHMT), a primarily symbolic asymmetrical approach that addresses the issue of Interlingual MT resource poverty in source-poor/target-rich language pairs by exploiting symbolic and statistical target-language resources. GHMT's statistical components are limited to target-language models, which arguably makes it a simple form of a hybrid system. We extend the hybrid nature of GHMT by adding statistical bilingual components. We also describe the details of retargeting it to Arabic-English MT. The morphological richness of Arabic brings several challenges to the hybridization task. We conduct an extensive evaluation of multiple system variants. Our evaluation shows that this new variant of GHMT-a primarily symbolic system extended with monolingual and bilingual statistical components-has a higher degree of grammaticality than a phrase-based statistical MT system, where grammaticality is measured in terms of correct verb-argument realization and long-distance dependency translation.

show abstract

Section: Morphology-based Approachessupporting

confidence: 53%

Symbolic-to-statistical hybridization: extending generation-heavy machine translation

Habash

Dorr

Monz

2009

Machine Translation

Self Cite

View full text Add to dashboard Cite

show abstract

“…Lee et al (2003) use a look-up table for the various prefixes, stems, and suffixes used in the tokenization process. Habash and Sadat (2006) presented various schemes for tokenizing Arabic text for MT, in addition to the Arabic Treebank tokenization (Maamouri et al, 2004). Diab et al (2007) presented an SVM-based approach for tokenization.…”

Section: Background and Related Workmentioning

confidence: 99%

“…However, especially for morphologically complex languages, using sub-lexical units obtained after morphological preprocessing has been shown to improve the machine translation performance over a wordbased system (Popović and Ney, 2004;Habash and Sadat, 2006). For any language, several word tokenization choices, henceforth tokenization schemes, can be generated based on the word's in-context morphological analysis.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

Zalmout

Habash

2017

The Prague Bulletin of Mathematical Linguistics

Self Cite

View full text Add to dashboard Cite

Tokenization is very helpful for Statistical Machine Translation (SMT), especially when translating from morphologically rich languages. Typically, a single tokenization scheme is applied to the entire source-language text and regardless of the target language. In this paper, we evaluate the hypothesis that SMT performance may benefit from different tokenization schemes for different words within the same text, and also for different target languages. We apply this approach to Arabic as a source language, with five target languages of varying morphological complexity: English, French, Spanish, Russian and Chinese. Our results show that different target languages indeed require different source-language schemes; and a context-variable tokenization scheme can outperform a context-constant scheme with a statistically significant performance enhancement of about 1.4 BLEU points.

show abstract

“…experiments [5], we used the MADA tool [12] for morphological disambiguation and applied the D2-scheme of [13] for word segmentation. These tools require the input to be Buckwalter encoded.…”

Section: Adjustment Of Asr and Smt Vocabulariesmentioning

confidence: 99%

The RWTH Arabic-to-English spoken language translation system

Bender

Matusov

Hahn

et al. 2007

2007 IEEE Workshop on Automatic Speech Recognition &Amp; Understanding (ASRU)

View full text Add to dashboard Cite

We present the RWTH phrase-based statistical machine translation system designed for the translation of Arabic speech into English text. This system was used in the Global Autonomous Language Exploitation (GALE) Go/No-Go Translation Evaluation 2007.Using a two-pass approach, we first generate n-best translation candidates and then rerank these candidates using additional models. We give a short review of the decoder as well as of the models used in both passes.We stress the difficulties of spoken language translation, i.e. how to combine the recognition and translation systems and how to compensate for missing punctuation. In addition, we cover our work on domain adaptation for the applied language models. We present translation results for the official GALE 2006 evaluation set and the GALE 2007 development set.

show abstract

Arabic preprocessing schemes for statistical machine translation

Cited by 115 publications

References 11 publications

Symbolic-to-statistical hybridization: extending generation-heavy machine translation

Symbolic-to-statistical hybridization: extending generation-heavy machine translation

Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

The RWTH Arabic-to-English spoken language translation system

Contact Info

Product

Resources

About