A Stochastic Decoder for Neural Machine Translation

Abstract

The process of translation is ambiguous, in that there are typically many valid translations for a given sentence. This gives rise to significant variation in parallel corpora, however, most current models of machine translation do not account for this variation, instead treating the problem as a deterministic process. To this end, we present a deep generative model of machine translation which incorporates a chain of latent variables, in order to account for local lexical and syntactic variation in parallel corpora. We provide an indepth analysis of the pitfalls encountered in variational inference for training deep generative models. Experiments on several different language pairs demonstrate that the model consistently improves over strong baselines. * Code and a workflow that reproduces the experiments are available at https://github.com/philschulz/ stochastic-decoder. † Work done prior to joining Amazon.

Latent Variable Model for Multi-modal Translation

“…Even though there has been growing interest in variational approaches to machine translation (Zhang et al, 2016;Schulz et al, 2018;Shah and Barber, 2018;Eikema and Aziz, 2019) and to tasks that integrate vision and language, e.g. image description generation (Pu et al, 2016;Wang et al, 2017), relatively little attention has been dedicated to variational models for multi-modal translation.…”

Auto-Encoding Variational Neural Machine Translation

“…Their formulation is a conditional deep generative model (Sohn et al, 2015) that does not model the source side of the data, where, rather than a fixed standard Gaussian, the latent model is itself parameterised and depends on the data. Schulz et al (2018) extend the model of with a Markov chain of latent variables, one per timestep, allowing the model to capture greater variability.…”

Sequence to Sequence Mixture Model for Diverse Machine Translation

“…Some work puts their efforts on decoding stages, and form a group of beam search to encourage diversity (Vijayakumar et al, 2016), while others pay more attention to adversarial training (Shetty et al, 2017;. Within translation, our method is similar to Schulz et al (2018b), where they propose a MT system armed with variational inference to account for translation variations. Like us, their diversified generation is driven by latent variables.…”

“…A well recognized issue with SEQ2SEQ models is the lack of diversity in the generated translations. This issue is mostly attributed to the decoding algorithm (Li et al, 2016), and recently to the model (Zhang et al, 2016;Schulz et al, 2018a). The former direction has attempted to design diversity encouraging decoding algorithm, particularly beam search, as it generates translations sharing the majority of their tokens except a few trailing ones.…”