Using Sublexical Translations to Handle the OOV Problem in Machine Translation

Abstract

We introduce a method for learning to translate out-of-vocabulary (OOV) words. The method focuses on combining sublexical/constituent translations of an OOV to generate its translation candidates. In our approach, wildcard searches are formulated based on our OOV analysis, aimed at maximizing the probability of retrieving OOVs’ sublexical translations from existing resources of Machine Translation (MT) systems. At run-time, translation candidates of the unknown words are generated from their suitable sublexical translations and ranked based on monolingual and bilingual information. We have incorporated the OOV model into a state-of-the-art machine translation system and experimental results show that our model indeed helps to ease the impact of OOVs on translation quality, especially for sentences containing more OOVs (significant improvement).

Statistical machine translation enhancements through linguistic levels

“…In the decoding time, if a numeral or temporal expression is found, it is substituted by the special symbol so that the surrounding words can be handled properly and finally the numeral/temporal expression is translated with the manually written rules. Huang et al [2011] propose a sublexical translation method to translate Chinese abbreviations and compounds. More recently, Zhang et al [2012] address the problem of the lexical selection and word reordering of the surrounding words caused by unknown words.…”

Sensible: L2 Translation Assistance by Emulating the Manual Post-Editing Process

“…Previous approaches to the handling of untranslated fragments include using a pivot language to translate the OOV word(s) into a third language and then back into to the source language, thereby extracting paraphrases to OOV (Callison-burch and Osborne, 2006), combining sub-lexical/constituent translations of the OOV word(s) to generate the translation (Huang et al, 2011) or finding paraphrases of the OOV words that have available translations (Marton et al, 2009;Razmara et al, 2013). 1 However the simplest approach to handle untranslated fragments is to increase the size of parallel data.…”

Reducing the Impact of Data Sparsity in Statistical Machine Translation

“…Untranslated out-of-vocabulary (OOV) words tend to degrade the accuracy of the output produced by an MT model. Huang (2010) pointed to various types of OOV words which occur in a data set -seg-mentation error in source language, named entities, combination forms (e.g. widebody) and abbreviations.…”