Improving Non-autoregressive Generation with Mixup Training

Jiang, Ting; Huang, Shaohan; Zhang, Zihan; Wang, Deqing; Zhuang, Fuzhen; Wei, Furu; Huang, Haizhen; Zhang, Liang-Jie; Zhang, Qi

doi:10.48550/arxiv.2110.11115

Cited by 5 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, we report the unigram ROUGE-1 and bigram ROUGE-2 overlap to assess the informativeness, and the longest common subsequence ROUGE-L score to assess the fluency. We compare our AMOM with the original CMLM and several NAR baseline models, including vanilla NAT (Gu et al 2018), InsertNAR (Stern et al 2019), Levenshitein (Gu, Wang, and Zhao 2019), Disco (Kasai et al 2020a), POSPD (Yang et al 2021), CMLM (Ghazvininejad et al 2019), BANG (Qi et al 2021), MIST (Jiang et al 2021), ELMER (Li et al 2022a). Results show that AMOM outperforms all other NAR models without pre-training.…”

Section: Resultsmentioning

confidence: 99%

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

Xiao¹,

Xu²,

et al. 2023

AAAI

View full text Add to dashboard Cite

Transformer-based autoregressive (AR) methods have achieved appealing performance for varied sequence-to-sequence generation tasks, e.g., neural machine translation, summarization, and code generation, but suffer from low inference efficiency. To speed up the inference stage, many non-autoregressive (NAR) strategies have been proposed in the past few years. Among them, the conditional masked language model (CMLM) is one of the most versatile frameworks, as it can support many different sequence generation scenarios and achieve very competitive performance on these tasks. In this paper, we further introduce a simple yet effective adaptive masking over masking strategy to enhance the refinement capability of the decoder and make the encoder optimization easier. Experiments on 3 different tasks (neural machine translation, summarization, and code generation) with 15 datasets in total confirm that our proposed simple method achieves significant performance improvement over the strong CMLM model. Surprisingly, our proposed model yields state-of-the-art performance on neural machine translation (34.62 BLEU on WMT16 EN to RO, 34.82 BLEU on WMT16 RO to EN, and 34.84 BLEU on IWSLT De to En) and even better performance than the AR Transformer on 7 benchmark datasets with at least 2.2x speedup. Our code is available at GitHub.

show abstract

Section: Resultsmentioning

confidence: 99%

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

Xiao¹,

Xu²,

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

“…The inference efficiency is not only required for neural machine translation but also indispensable for many other text generation tasks [81], [100], [101]. Existing works of introducing NAR techniques into text generation tasks focus on automatic speech recognition [102], [103], [104], text summarization [105], grammatical error correction [106], [107], dialogue [108], [109].…”

Section: Text Generationmentioning

confidence: 99%

“…to enhance auto-regressive models with powerful pretraining techniques and models, with impressive performance being achieved. However, only very few papers apply these powerful pre-trained models to help NAR models [80], [101], and there is only a preliminary exploration of the pre-training techniques for NAR models [40], [105]. Thus, it is promising to explore pretraining methods for non-autoregressive generation and other related tasks.…”

Section: Conclusion and Outlooksmentioning

confidence: 99%

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Xiao¹,

Wu²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, auto-regressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as dialogue generation, text summarization, grammar error correction, semantic parsing, speech synthesis, and automatic speech recognition. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, dynamic length prediction, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications.

show abstract

“…Text generation can also be done with masked-language models, e.g. BERT [Su et al, 2021, Jiang et al, 2021. Apart from left to right text generation, estimating sentence/word probabilities from masked language models is also useful for various NLP tasks.…”

Section: Mlmmentioning

confidence: 99%

Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Models

Zouhar¹,

Mosbach²,

Klakow³

2022

Preprint

View full text Add to dashboard Cite

Although masked language models are highly performant and widely adopted by NLP practitioners, they can not be easily used for autoregressive language modelling (next word prediction and sequence probability estimation). We present an LSTM-based autoregressive language model which uses prefix embeddings (from a pretrained masked language model) via fusion (e.g. concatenation) to obtain a richer context representation for language modelling. We find that fusion helps reliably in lowering the perplexity (16.74 → 15.80), which is even preserved after a transfer to a dataset from a different domain than the training data. We also evaluate the best-performing fusion model by correlating its next word surprisal estimates with human reading times. Contradicting our expectation, and despite the improvement in perplexity overall, the correlation remains the same as for the baseline model. Lastly, while we focus on language models pre-trained on text as the sources for the fusion, our approach can be possibly extended to fuse any information represented as a fixed-size vector into an auto-regressive language model. These include e.g. sentence external information retrieved for a knowledge base or representations of multi-modal encoders.

show abstract

Improving Non-autoregressive Generation with Mixup Training

Cited by 5 publications

References 29 publications

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Models

Contact Info

Product

Resources

About