Target Conditioning for One-to-Many Generation

Abstract

Neural Machine Translation (NMT) models often lack diversity in their generated translations, even when paired with search algorithm, like beam search. A challenge is that the diversity in translations are caused by the variability in the target language, and cannot be inferred from the source sentence alone. In this paper, we propose to explicitly model this oneto-many mapping by conditioning the decoder of a NMT model on a latent variable that represents the domain of target sentences. The domain is a discrete variable generated by a target encoder that is jointly trained with the NMT model. The predicted domain of target sentences are given as input to the decoder during training. At inference, we can generate diverse translations by decoding with different domains. Unlike our strongest baseline (Shen et al., 2019), our method can scale to any number of domains without affecting the performance or the training time. We assess the quality and diversity of translations generated by our model with several metrics, on three different datasets.

Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

“…Generating multiple valid outputs given a source sequence has a wide range of applications, such as machine translation (Shen et al, 2019), paraphrase generation (Gupta et al, 2018), question generation (Cho et al, 2019), dialogue system (Dou et al, 2021), and story generation . For example, in machine translation, there are often many plausible and semantically equivalent translations due to information asymmetry between different languages (Lachaux et al, 2020).…”

“…For example, nucleus sampling samples next tokens from the dynamic nucleus of tokens containing the vast majority of the probability mass, instead of decoding text by maximizing the likelihood. Another line of work focused on introducing random noise (Gupta et al, 2018) or changing latent variables (Lachaux et al, 2020) to produce uncertainty. In addition, Shen et al (2019) adopted a mixture of experts to diversify machine translation, where a minimum-loss predictor is assigned to each source input.…”

Bridging the Gap between Training and Inference: Multi-Candidate Optimization for Diverse Neural Machine Translation

“…Further, Lachaux et al (2020) replace the syntactic codes with latent domain variables derived from target sentences, which is more computationally efficient. Sun et al (2020) sample the encoderdecoder attention heads of Transformer to affect source word selection.…”

Sentence-Permuted Paragraph Generation

“…Diverse Text Generation. Generating diverse sequences is of crucial importance in many text generation applications that exhibit semantically oneto-many relationships between source and the target sequences, such as machine translation (Shen et al, 2019;Lachaux et al, 2020), summarization (Cho et al, 2019), question generation , and paraphrase generation (Qian et al, 2019). Methods of improving diversity in text generation that have been widely explored from different perspectives in recent years.…”

“…Samplingbased decoding is one of the effective solutions to improve diversity (Fan et al, 2018;Holtzman et al, 2020), e.g., nucleus sampling (Holtzman et al, 2020) samples next tokens from the dynamic nucleus of tokens containing the vast majority of the probability mass, instead of aiming to decode text by maximizing the likelihood. Another line of work focuses on introducing random noise (Gupta et al, 2018) or changing latent variable (Lachaux et al, 2020) to produce uncertainty, e.g., Gupta et al (2018) employ a variational auto-encoder framework to generate diverse paraphrases according to the input noise. In addition, Shen et al (2019) adopt a deep mixture of experts (MoE) to diversify machine translation, where a minimum-loss predictor is assigned to each source input; Shi et al (2018) employ inverse reinforcement learning for unconditional diverse text generation.…”