Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism

Xie, Pan; Zhi, Cui; Chen, Xiuying; Hu, Xiaohui; Cui, Jianwei; Wang, Bin

doi:10.18653/v1/2020.coling-main.2

Cited by 10 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2) introducing self-review mechanism [87], which applies an AR-decoder to help infuse sequential information; (3) selecting masked tokens with advanced strategies [26], [34], [88]. Geng et al [26] focus on the importance of determining the tokens replaced by [mask] tokens in next iteration and propose a revisor and locator for rewriting.…”

Section: Encoder Stackmentioning

confidence: 99%

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Xiao¹,

Wu²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, auto-regressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as dialogue generation, text summarization, grammar error correction, semantic parsing, speech synthesis, and automatic speech recognition. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, dynamic length prediction, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications.

show abstract

Section: Encoder Stackmentioning

confidence: 99%

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Xiao¹,

Wu²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Headline Generation. In recent years, text generation has made impressive progress (Li et al 2019;Chan et al 2019;Liu et al 2020;Xie et al 2020;Chan et al 2020;Chen et al 2021), and headline generation has become a research hotspot in Natural Language Processing. Most existing headline generation works solely focus on summarizing the document.…”

Section: Related Workmentioning

confidence: 99%

The Style-Content Duality of Attractiveness: Learning to Write Eye-Catching Headlines via Disentanglement

Chen

Yang

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

Eye-catching headlines function as the first device to trigger more clicks, bringing reciprocal effect between producers and viewers. Producers can obtain more traffic and profits, and readers can have access to outstanding articles. When generating attractive headlines, it is important to not only capture the attractive content but also follow an eye-catching writtenstyle. In this paper, we propose a Disentanglement-based Attractive Headline Generator (DAHG) that generates headline which captures the attractive content following the attractive style. Concretely, we first devise a disentanglement module to divide the style and content of an attractive prototype headline into latent spaces, with two auxiliary constraints to ensure the two spaces are indeed disentangled. The latent content information is then used to further polish the document representation and help capture the salient part. Finally, the generator takes the polished document as input to generate headline under the guidance of the attractive style. Extensive experiments on the public Kuaibao dataset show that DAHG achieves state-of-the-art performance. Human evaluation also demonstrates that DAHG triggers 22% more clicks than existing models.

show abstract

“…Multi-turn Dialog. In recent years, text generation has made impressive progress (Li et al 2018;Chan et al 2019;Gao et al 2020b;Xie et al 2020), and multi-turn dialog model aims to take a message and utterances in previous turns as input and generates a response (Tao et al 2019;Gao et al 2020a). Several works (Zhang et al 2019;Adiwardana et al 2020;Chan et al 2020) simplify the multi-turn dialog into single-turn problem by simply concatenating multiple sentences into one sentence, and utilized the basic Seq2seq based on RNN or Transformer to model long sequence.…”

Section: Related Workmentioning

confidence: 99%

Reasoning in Dialog: Improving Response Generation by Context Reading Comprehension

Chen

Zhi

Zhang

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

In multi-turn dialog, utterances do not always take the full form of sentences (Carbonell 1983), which naturally makes understanding the dialog context more difficult. However, it is essential to fully grasp the dialog context to generate a reasonable response. Hence, in this paper, we propose to improve the response generation performance by examining the model's ability to answer a reading comprehension question, where the question is focused on the omitted information in the dialog. Enlightened by the multi-task learning scheme, we propose a joint framework that unifies these two tasks, sharing the same encoder to extract the common and task-invariant features with different decoders to learn task-specific features. To better fusing information from the question and the dialog history in the encoding part, we propose to augment the Transformer architecture with a memory updater, which is designed to selectively store and update the history dialog information so as to support downstream tasks. For the experiment, we employ human annotators to write and examine a large-scale dialog reading comprehension dataset. Extensive experiments are conducted on this dataset, and the results show that the proposed model brings substantial improvements over several strong baselines on both tasks. In this way, we demonstrate that reasoning can indeed help better response generation and vice versa. We release our large-scale dataset for further research.

show abstract

Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism

Cited by 10 publications

References 12 publications

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

The Style-Content Duality of Attractiveness: Learning to Write Eye-Catching Headlines via Disentanglement

Reasoning in Dialog: Improving Response Generation by Context Reading Comprehension

Contact Info

Product

Resources

About