Character-level Transformer-based Neural Machine Translation

Abstract

Neural machine translation (NMT) is nowadays commonly applied at the subword level, using byte-pair encoding. A promising alternative approach focuses on character-level translation, which simplifies processing pipelines in NMT considerably. This approach, however, must consider relatively longer sequences, rendering the training process prohibitively expensive. In this paper, we discuss a novel, Transformer-based approach, that we compare, both in speed and in quality to the Transformer at subword and character levels, as well as previously developed character-level models. We evaluate our models on 4 language pairs from WMT'15: DE-EN, CS-EN, FI-EN and RU-EN. The proposed novel architecture can be trained on a single GPU and is ∼34% faster than the character-level Transformer; still, the obtained results are at least on par with it. In addition, our proposed model outperforms the subword-level model in FI-EN and shows close results in CS-EN. To stimulate further research in this area and close the gap with subword-level NMT, we make all our code and models publicly available.

When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation

“…However, translating at character level may incur significant computational overhead. Therefore, later works on character-level NMT (Cherry et al, 2018;Banar et al, 2020) mainly focus on reducing computation cost of them. Cherry et al (2018) show that by employing source sequence compression techniques, the quality and efficiency of character-based models can be properly balanced.…”

“…Cherry et al (2018) show that by employing source sequence compression techniques, the quality and efficiency of character-based models can be properly balanced. Banar et al (2020) share the same idea as Cherry et al (2018) but build their models using Transformer architecture. Our work differs from theirs in that we aim to analyze the performance of existing models instead of exploring novel architectures.…”

“…An alternative segmentation choice is to use fully character-level (CHAR) models (Lee et al, 2017;Cherry et al, 2018;Gupta et al, 2019;Gao et al, 2020;Banar et al, 2020), which has the potential to alleviate above issues. CHAR does not need to learn any segmentation rules and keeps all available information in the surface form, avoiding the risk of information loss due to improper segmentation.…”