Evaluating Discourse Phenomena in Neural Machine Translation

Abstract

For machine translation to tackle discourse phenomena, models must have access to extrasentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models' ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multiencoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.

One Model to Learn Both: Zero Pronoun Prediction and Translation

“…Recent studies revealed that inter-sentential context can implicitly help to tackle anaphora resolution in NMT architecture (Jean et al, 2017b;Bawden et al, 2018;Voita et al, 2018). Some may argue that document-level architectures are strong enough to alleviate ZP problems for NMT.…”

Document-Level Neural Machine Translation with Hierarchical Attention Networks

“…By ignoring discourse connections between sentences and other valuable contextual information, this simplification potentially degrades the coherence and cohesion of a translated document (Hardmeier, 2012;Meyer and Webber, 2013;Sim Smith, 2017). Recent studies (Tiedemann and Scherrer, 2017;Wang et al, 2017; have demonstrated that adding contextual information to the NMT model improves the general translation performance, and more importantly, improves the coherence and cohesion of the translated text (Bawden et al, 2018;Lapshinova-Koltunski and Hardmeier, 2017). Most of these methods use an additional encoder Wang et al, 2017) to extract contextual information from previous source-side sentences.…”

Improving the Transformer Translation Model with Document-Level Context

“…Developing document-level models for machine translation has been an important research direction, both for conventional SMT (Gong et al, 2011;Hardmeier et al, 2012;Xiong et al, 2013a,b;Garcia et al, 2014) and NMT (Jean et al, 2017;Kuang et al, 2017;Tiedemann and Scherrer, 2017;Wang et al, 2017;Maruf and Haffari, 2018;Bawden et al, 2018;Tu et al, 2018;Voita et al, 2018).…”

“…While document-level NMT has attracted increasing attention from the community in the past two years (Jean et al, 2017;Kuang et al, 2017;Tiedemann and Scherrer, 2017;Wang et al, 2017;Maruf and Haffari, 2018;Bawden et al, 2018;Tu et al, 2018;Voita et al, 2018), to the best of our knowledge, only one existing work has endeavored to model document-level context for the Transformer model (Voita et al, 2018). Previous approaches to document-level NMT have concentrated on the RNNsearch model (Bahdanau et al, 2015).…”