When and Why is Document-level Context Useful in Neural Machine Translation?

Abstract

Document-level context has received lots of attention for compensating neural machine translation (NMT) of isolated sentences. However, recent advances in document-level NMT focus on sophisticated integration of the context, explaining its improvement with only a few selected examples or targeted test sets. We extensively quantify the causes of improvements by a document-level model in general test sets, clarifying the limit of the usefulness of document-level context in NMT. We show that most of the improvements are not interpretable as utilizing the context. We also show that a minimal encoding is sufficient for the context modeling and very long context is not helpful for NMT.

Tencent Neural Machine Translation Systems for WMT18

“…The interesting finding is that the document-level model trained on pseudo contexts ("IN+OUT") can improve the baseline that is trained on only real context ("IN") by +5.47 BLEU points. We think there are two main reasons: 1) it lacks of large-scale training data with contextual information; 2) it is still unclear how the context help document translation, which is similar to the conclusion in previous work (Kim et al, 2019;. About NAT models, our proposed approach can improve the vanilla NAT by +0.6 BLEU point, which are lower than those of autoregressive NMT models.…”

“…We and we use " /s " symbols as their pseudo contexts (Kim et al, 2019;. Besides, we conduct Sent-Out→Doc-In finetuning (different architectures and data).…”

Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learning

“…They ignore the individualized needs for context when translating different source sentences. Some works have noticed that not all context is useful (Jean and Cho, 2019;Kim et al, 2019). Kimura et al (2019) explore the context selection in the single-encoder framework (Tiedemann and Scherrer, 2017), and select context sentences that yield highest forced back-translation probability.…”

Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation

“…For a fair evaluation of context-aware NMT methods, we argue that one should build a strong enough sentence-level baseline with carefully regularized methods, especially on small datasets (Kim et al, 2019;Sennrich and Zhang, 2019). Beyond this, Bawden et al (2018) and Voita et al (2019) acknowledged that BLEU score is insufficient to evaluate context-aware models, and they emphasized that multi-encoder architectures alone had a limited capacity to exploit discourse-level context.…”