Correcting Length Bias in Neural Machine Translation

Murray, Kenton; Chiang, David

doi:10.18653/v1/w18-6322

Cited by 120 publications

(106 citation statements)

References 17 publications

Supporting

Mentioning

102

Contrasting

Order By: Relevance

“…The length ratio is not just about BLEU: if the hypothesis length is only 75% of reference length, something that should have been translated must be missing; i.e., bad adequacy. Indeed,Murray and Chiang (2018) confirm the same phenomenon with METEOR.2 Pre-neural SMT models, being probabilistic, also favor short translations (and derivations), which is addressed by word (and phrase) reward. The crucial difference between SMT and NMT is that the former stops when covering the whole input, while the latter stops on emitting </eos>.…”

mentioning

confidence: 54%

See 1 more Smart Citation

Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation

Yang¹,

Huang²,

Ma³

2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Beam search is widely used in neural machine translation, and usually improves translation quality compared to greedy search. It has been widely observed that, however, beam sizes larger than 5 hurt translation quality. We explain why this happens, and propose several methods to address this problem. Furthermore, we discuss the optimal stopping criteria for these methods. Results show that our hyperparameter-free methods outperform the widely-used hyperparameter-free heuristic of length normalization by +2.0 BLEU, and achieve the best results among all methods on Chinese-to-English translation.

show abstract

mentioning

confidence: 54%

“…Murray and Chiang (2018) attribute the fact that beam search prefers shorter candidates to the label bias problem(Lafferty et al, 2001) due to NMT's local normalization.…”

mentioning

confidence: 99%

Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation

Yang¹,

Huang²,

Ma³

2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…A potential confound is that performance might change with the length of the source in BiLSTMs (Carpuat et al, 2013;Murray and Chiang, 2018), in Transformers it was reported to increase . Length is generally greater in the challenge set than in the full test set, and generally increases with d, showing if anything a decrease of performance by length.…”

Section: Methodsmentioning

confidence: 99%

Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

Choshen¹,

Abend²

2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

We show that the state-of-the-art Transformer MT model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, longdistance dependencies remain a challenge for the model. Since most dependencies are shortdistance, common evaluation metrics will be little influenced by how well systems perform on them. We therefore propose an automatic approach for extracting challenge sets replete with long-distance dependencies, and argue that evaluation using this methodology provides a complementary perspective on system performance. To support our claim, we compile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow reliable automatic evaluation, which makes the proposed approach a scalable and practical solution for evaluating MT performance on the long-tail of syntactic phenomena. 1

show abstract

“…This difficulty has been observed in datasets designed to test the ability of models to compositionally generalize, such as SCAN , where the best performing neural models do not even exceed 20% accuracy on generating sequences of out-of-domain lengths, whereas indomain performance is 100%. Extrapolation has also been a challenge for neural machine translation; Murray and Chiang (2018) identifies models producing translations that are too short as one of the main challenges for neural MT.…”

Section: Related Workmentioning

confidence: 99%

The EOS Decision and Length Extrapolation

Newman

Hewitt

Liang

et al. 2020

Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

Extrapolation to unseen sequence lengths is a challenge for neural generative models of language. In this work, we characterize the effect on length extrapolation of a modeling decision often overlooked: predicting the end of the generative process through the use of a special end-of-sequence (EOS) vocabulary item. We study an oracle setting-forcing models to generate to the correct sequence length at test time-to compare the lengthextrapolative behavior of networks trained to predict EOS (+EOS) with networks not trained to (-EOS). We find that -EOS substantially outperforms +EOS, for example extrapolating well to lengths 10 times longer than those seen at training time in a bracket closing task, as well as achieving a 40% improvement over +EOS in the difficult SCAN dataset length generalization task. By comparing the hidden states and dynamics of -EOS and +EOS models, we observe that +EOS models fail to generalize because they (1) unnecessarily stratify their hidden states by their linear position is a sequence (structures we call length manifolds) or (2) get stuck in clusters (which we refer to as length attractors) once the EOS token is the highest-probability prediction.

show abstract

Correcting Length Bias in Neural Machine Translation

Cited by 120 publications

References 17 publications

Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation

Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation

Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

The EOS Decision and Length Extrapolation

Contact Info

Product

Resources

About