Sequence to Sequence Mixture Model for Diverse Machine Translation

He, Xuanli; Haffari, Gholamreza; Norouzi, Mohammad

doi:10.18653/v1/k18-1056

Cited by 41 publications

(29 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Li et al (2016); Vijayakumar et al (2016) adjust decoding algorithms, adding different kinds of diversity regularization terms to encourage generating diverse outputs. He et al (2018); Shen et al (2019) utilize mixture of experts (MoE) method, using differentiated latent variables to control generation of translation. Sun et al (2019) generates diverse translation by sampling heads in encoder-decoder attention module in Transformer model, since different heads may present different target-source alignment.…”

Section: Analyzing Module Importance Withmentioning

confidence: 99%

“…Li et al (2016) and Vijayakumar et al (2016) proposed to add regularization terms to the beam search algorithm so that it can possess greater diversity. He et al (2018) and Shen et al (2019) introduced latent variables into the NMT model, thus the model can generate diverse outputs using different latent variables. Moreover, Sun et al (2019) proposed to combine the structural characteristics of Transformer and use the different weights between each head in the multi-head attention mechanism to obtain diverse results.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Generating Diverse Translation from Model Distribution with Dropout

Wu¹,

Feng

Shao

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Despite the improvement of translation quality, neural machine translation (NMT) often suffers from the lack of diversity in its generation. In this paper, we propose to generate diverse translations by deriving a large number of possible models with Bayesian modelling and sampling models from them for inference. The possible models are obtained by applying concrete dropout to the NMT model and each of them has specific confidence for its prediction, which corresponds to a posterior model distribution under specific training data in the principle of Bayesian modeling. With variational inference, the posterior model distribution can be approximated with a variational distribution, from which the final models for inference are sampled. We conducted experiments on Chinese-English and English-German translation tasks and the results shows that our method makes a better trade-off between diversity and accuracy.

show abstract

Section: Analyzing Module Importance Withmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Generating Diverse Translation from Model Distribution with Dropout

Wu¹,

Feng

Shao

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…One of the most successful applications of MoE is ensemble learning (Caruana et al, 2004;Liu et al, 2018;Dutt et al, 2017, inter alia). Recent efforts also explore MoE in sequence learning , and to promote diversity in text generation (He et al, 2018;Shen et al, 2019;Cho et al, 2019, inter alia).…”

Section: Related Workmentioning

confidence: 99%

A Mixture of h - 1 Heads is Better than h Heads

Peng¹,

Schwartz²,

Li³

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Multi-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks. Evidence has shown that they are overparameterized; attention heads can be pruned without significant performance loss. In this work, we instead "reallocate" them-the model learns to activate different heads on different inputs. Drawing connections between multi-head attention and mixture of experts, we propose the mixture of attentive experts model (MAE). MAE is trained using a block coordinate descent algorithm that alternates between updating (1) the responsibilities of the experts and (2) their parameters. Experiments on machine translation and language modeling show that MAE outperforms strong baselines on both tasks. Particularly, on the WMT14 English to German translation dataset, MAE improves over "transformer-base" by 0.8 BLEU, with a comparable number of parameters. Our analysis shows that our model learns to specialize different experts to different inputs. 1

show abstract

“…Prior work on diverse machine translation (He et al, 2018a;Shen et al, 2019;Cho et al, 2019) suggests to apply online hard EM by interleaving the following two steps for each mini-batch.…”

Section: Kb Module For Inferential Reasoningmentioning

confidence: 99%

CosMo: Conditional Seq2Seq-based Mixture Model for Zero-Shot Commonsense Question Answering

Moghimifar

Yang³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Commonsense reasoning refers to the ability of evaluating a social situation and acting accordingly. Identification of the implicit causes and effects of a social context is the driving capability which can enable machines to perform commonsense reasoning. The dynamic world of social interactions requires context-dependent on-demand systems to infer such underlying information. However, current approaches in this realm lack the ability to perform commonsense reasoning upon facing an unseen situation, mostly due to incapability of identifying a diverse range of implicit social relations. Hence they fail to estimate the correct reasoning path. In this paper, we present Conditional SEQ2SEQ-based Mixture model (COSMO), which provides us with the capabilities of dynamic and diverse content generation. We use COSMO to generate context-dependent clauses, which form a dynamic Knowledge Graph (KG) on-the-fly for commonsense reasoning. To show the adaptability of our model to context-dependant knowledge generation, we address the task of zero-shot commonsense question answering. The empirical results indicate an improvement of up to +5.2% over the state-of-the-art models.

show abstract

Sequence to Sequence Mixture Model for Diverse Machine Translation

Cited by 41 publications

References 22 publications

Generating Diverse Translation from Model Distribution with Dropout

Generating Diverse Translation from Model Distribution with Dropout

A Mixture of h - 1 Heads is Better than h Heads

CosMo: Conditional Seq2Seq-based Mixture Model for Zero-Shot Commonsense Question Answering

Contact Info

Product

Resources

About