Context-Aware Neural Machine Translation Learns Anaphora Resolution

Voita, Elena; Serdyukov, Pavel; Sennrich, Rico; Titov, Ivan

doi:10.18653/v1/p18-1117

Cited by 231 publications

(293 citation statements)

References 13 publications

Supporting

Mentioning

283

Contrasting

Order By: Relevance

“…There are several works analyzing attention weights of different NMT models (Ghader and Monz, 2017;Voita et al, 2018;Raganato and Tiedemann, 2018). Raganato and Tiedemann (2018) use the self-attention weights of the Transformer's encoder to induce a tree structure for each sentence and compute the unlabeled attachment score of these trees.…”

Section: Related Workmentioning

confidence: 99%

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Voita¹,

Talbot²,

Moiseev³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Self Cite

728

544

View full text Add to dashboard Cite

Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by individual attention heads in the encoder to the overall performance of the model and analyze the roles played by them. We find that the most important and confident heads play consistent and often linguistically-interpretable roles. When pruning heads using a method based on stochastic gates and a differentiable relaxation of the L 0 penalty, we observe that specialized heads are last to be pruned. Our novel pruning method removes the vast majority of heads without seriously affecting performance. For example, on the English-Russian WMT dataset, pruning 38 out of 48 encoder heads results in a drop of only 0.15 BLEU. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Voita¹,

Talbot²,

Moiseev³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Self Cite

728

544

View full text Add to dashboard Cite

show abstract

“…Recent studies revealed that inter-sentential context can implicitly help to tackle anaphora resolution in NMT architecture (Jean et al, 2017b;Bawden et al, 2018;Voita et al, 2018). Some may argue that document-level architectures are strong enough to alleviate ZP problems for NMT.…”

Section: Effect Of Discourse-level Contextmentioning

confidence: 99%

One Model to Learn Both: Zero Pronoun Prediction and Translation

Wang¹,

Tu²,

Wang³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Zero pronouns (ZPs) are frequently omitted in pro-drop languages, but should be recalled in non-pro-drop languages. This discourse phenomenon poses a significant challenge for machine translation (MT) when translating texts from pro-drop to non-pro-drop languages. In this paper, we propose a unified and discourseaware ZP translation approach for neural MT models. Specifically, we jointly learn to predict and translate ZPs in an end-to-end manner, allowing both components to interact with each other. In addition, we employ hierarchical neural networks to exploit discourse-level context, which is beneficial for ZP prediction and thus translation. Experimental results on both Chinese⇒English and Japanese⇒English data show that our approach significantly and accumulatively improves both translation performance and ZP prediction accuracy over not only baseline but also previous works using external ZP prediction models. Extensive analyses confirm that the performance improvement comes from the alleviation of different kinds of errors especially caused by subjective ZPs.

show abstract

“…Though this problem has recently triggered a lot of attention to contextaware translation (Jean et al, 2017a;Wang et al, 2017;Tiedemann and Scherrer, 2017; Bawden 1 We release code and data sets at https://github.com/lena-voita/ good-translation-wrong-in-context. Voita et al, 2018;Maruf and Haffari, 2018;Agrawal et al, 2018;Miculicich et al, 2018;Zhang et al, 2018), the progress and widespread adoption of the new paradigm is hampered by several important problems. Firstly, it is highly non-trivial to design metrics which would reliably trace the progress and guide model design.…”

Section: Introductionmentioning

confidence: 99%

When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion

Voita¹,

Sennrich²,

Titov³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Self Cite

150

185

View full text Add to dashboard Cite

Though machine translation errors caused by the lack of context beyond one sentence have long been acknowledged, the development of context-aware NMT systems is hampered by several problems. Firstly, standard metrics are not sensitive to improvements in consistency in document-level translations. Secondly, previous work on context-aware NMT assumed that the sentence-aligned parallel data consisted of complete documents while in most practical scenarios such document-level data constitutes only a fraction of the available parallel data. To address the first issue, we perform a human study on an English-Russian subtitles dataset and identify deixis, ellipsis and lexical cohesion as three main sources of inconsistency. We then create test sets targeting these phenomena. To address the second shortcoming, we consider a set-up in which a much larger amount of sentence-level data is available compared to that aligned at the document level. We introduce a model that is suitable for this scenario and demonstrate major gains over a context-agnostic baseline on our new benchmarks without sacrificing performance as measured with BLEU. 1 1. ambiguity arising when imperative and indicative verb forms are the same,

show abstract

Context-Aware Neural Machine Translation Learns Anaphora Resolution

Cited by 231 publications

References 13 publications

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

One Model to Learn Both: Zero Pronoun Prediction and Translation

When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion

Contact Info

Product

Resources

About