On the Importance of Word Boundaries in Character-level Neural Machine Translation

Abstract

Neural Machine Translation (NMT) models generally perform translation using a fixedsize lexical vocabulary, which is an important bottleneck on their generalization capability and overall translation quality. The standard approach to overcome this limitation is to segment words into subword units, typically using some external tools with arbitrary heuristics, resulting in vocabulary units not optimized for the translation task. Recent studies have shown that the same approach can be extended to perform NMT directly at the level of characters, which can deliver translation accuracy on-par with subword-based models, on the other hand, this requires relatively deeper networks. In this paper, we propose a more computationally-efficient solution for character-level NMT which implements a hierarchical decoding architecture where translations are subsequently generated at the level of words and characters. We evaluate different methods for open-vocabulary NMT in the machine translation task from English into five languages with distinct morphological typology, and show that the hierarchical decoding model can reach higher translation accuracy than the subword-level NMT model using significantly fewer parameters, while demonstrating better capacity in learning longer-distance contextual and grammatical dependencies than the standard character-level NMT model.

Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems

“…On the other hand, the characterlevel models perform worse on compounds, which are a local phenomenon. Ataman et al (2019) observed similar results on compounds in their hybrid character-word-level method. We suspect this might be caused by poor memorization of some compounds in the character models.…”

“…Cherry et al (2018) showed that with a sufficiently deep recurrent model, no changes in the model are necessary, and they can still reach translation quality that is on par with subword models. Luong and Manning (2016) and Ataman et al (2019) can leverage characterlevel information but they require tokenized text as an input and only have access to the character-level embeddings of predefined tokens.…”

Revisiting Negation in Neural Machine Translation

“…More specifically, the deletion of negation cues causes more errors. Ataman et al (2019) show that character-level models perform better than subword-level models on negation. Instead, we evaluate NMT models with different neural networks to learn their abilities to translate negation, by scoring contrastive translate pairs.…”

An Effective Conversion of Visemes to Words for High-Performance Automatic Lipreading

“…The second reason for there being a better resilience is that the GRU network is better at modelling shorter groups of words [ 51 ]. It does not suffer from the problem posed in the mapping of viseme clusters to words using the GPT-based iterator whereby compound errors occur in the combination of words during the iterations and in which the sentence being decoded is based on the conditional dependence of word combinations.…”