Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior

Abstract

Although neural machine translation models reached high translation quality, the autoregressive nature makes inference difficult to parallelize and leads to high translation latency. Inspired by recent refinement-based approaches, we propose LaNMT, a latent-variable non-autoregressive model with continuous latent variables and deterministic inference procedure. In contrast to existing approaches, we use a deterministic inference algorithm to find the target sequence that maximizes the lowerbound to the log-probability. During inference, the length of translation automatically adapts itself. Our experiments show that the lowerbound can be greatly increased by running the inference algorithm, resulting in significantly improved translation quality. Our proposed model closes the performance gap between non-autoregressive and autoregressive approaches on ASPEC Ja-En dataset with 8.6x faster decoding. On WMT'14 En-De dataset, our model narrows the gap with autoregressive baseline to 2.0 BLEU points with 12.5x speedup. By decoding multiple initial latent variables in parallel and rescore using a teacher model, the proposed model further brings the gap down to 1.0 BLEU point on WMT'14 En-De task with 6.8x speedup.

FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

“…We first conduct experiments to compare the performance of FlowSeq with strong baseline models, including NAT w/ Fertility (Gu et al, 2018), NAT-IR , NAT-REG (Wang et al, 2019), LV NAR (Shu et al, 2019), CTC Loss (Libovickỳ and Helcl, 2018), and CMLM (Ghazvininejad et al, 2019).…”

Recursive Top-Down Production for Sentence Generation with Latent Trees

“…Our baseline models include an LSTM sequenceto-sequence with attention, Transformer (Vaswani et al, 2017), and a non-autoregressive model LaNMT (Shu et al, 2020). For a fair comparison, we trained all models with negative loglikelihood loss or knowledge distillation (Kim and Rush, 2016) if applicable.…”

Masked Graph Modeling for Molecule Generation

“…Latent variable models such as variational autoencoders and adversarial autoencoders assume the existence of unobserved (latent) variables Z = {z 1 , z 2 , ..., z k } that aim to capture dependencies among the vertices V and edges E of a graph G. Unlike an autoregressive model, a latent variable model does not necessarily require a predefined ordering of the graph [14]. The generation process consists of first sampling latent variables according to their prior distributions, followed by sampling vertices and edges conditioned on these latent variable samples.…”