Pretrained Language Models for Text Generation: A Survey

Li, Junyi; Tang, Tianyi; Zhao, Wayne Xin; Nie, Jian‐Yun; Wen, Ji-Rong

doi:10.48550/arxiv.2201.05273

Cited by 21 publications

(21 citation statements)

References 150 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. , y n ) conditioned on input data X (e.g., one or more pieces of text and structured data) [78]. Typically, NLG tasks are categorized according to the data format of X and Y .…”

Section: Data Collectionmentioning

confidence: 99%

MVP: Multi-task Supervised Pre-training for Natural Language Generation

Tang¹,

Li²,

Zhao³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have achieved notable success in natural language generation (NLG) tasks. Up to now, most of the PLMs are pre-trained in an unsupervised manner using large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with less labeled data showcase superior performance compared to unsupervised models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. For pre-training the text generation model MVP, we collect a labeled pre-training corpus from 45 datasets over seven generation tasks. For each task, we further pre-train specific soft prompts to stimulate the model capacity in performing a specific task. Extensive experiments have demonstrated the effectiveness of our supervised pre-training in a number of NLG tasks, and our general methods achieve state-of-the-art performance on 12 of 17 datasets.

show abstract

“…. , y n ) conditioned on input data X (e.g., one or more pieces of text and structured data) [78]. Typically, NLG tasks are categorized according to the data format of X and Y .…”

Section: Data Collectionmentioning

confidence: 99%

MVP: Multi-task Supervised Pre-training for Natural Language Generation

Tang¹,

Li²,

Zhao³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The main goal of our work is to improve the global diversity of questions generated by an underlying pre-trained Transformer [18] model. Most existing works have fine-tuned pre-trained Transformer models with the primary objective of maximizing question generation likelihood; while diversity was commonly dealt only as a secondary objective during inference time [8,23]. Compared to [9], which have utilized the Transformer model for the same task, we do not use any auxiliary data.…”

Section: Main Differencesmentioning

confidence: 99%

“…To this end, using a SentenceTransformer [14] dedicated for paraphrasing tasks, for each product category, we obtain embeddings for the test-set generated questions of both T5 and T5+LTD. Next, using cosine similarity as the "distance" metric, we obtain question clusters 8 for each alternative and measure the number of clusters obtained for increasing (dissimilarity) thresholds. We report the results in Figure 2, illustrating the relative improvement (green bars) or degradation (red bars) of T5+LTD compared to T5.…”

Section: Rq3mentioning

confidence: 99%

See 1 more Smart Citation

Learning to Diversify for Product Question Generation

Roitman¹,

Singer²,

Eshel³

et al. 2022

Preprint

View full text Add to dashboard Cite

We address the product question generation task. For a given product description, our goal is to generate questions that reflect potential user information needs that are either missing or not well covered in the description. Moreover, we wish to cover diverse user information needs that may span a multitude of product types.To this end, we first show how the T5 pre-trained Transformer encoder-decoder model can be fine-tuned for the task. Yet, while the T5 generated questions have a reasonable quality compared to the state-of-the-art method for the task (KPCNet), many of such questions are still too general, resulting in a sub-optimal global question diversity. As an alternative, we propose a novel learningto-diversify (LTD) fine-tuning approach that allows to enrich the language learned by the underlying Transformer model. Our empirical evaluation shows that, using our approach significantly improves the global diversity of the underlying Transformer model, while preserves, as much as possible, its generation relevance.

show abstract

“…The recent progress with Language Models (LM) in Natural Language Processing (NLP) has made it even easier, cheaper, and faster for a machine to generate artificial text [3]. Generative LMs have become sophisticated such as GPT-3 at text generation, and their results are almost on par with humanwritten text in terms of readability, coherence, and grammatical accuracy [4].…”

Section: Introductionmentioning

confidence: 99%

ExClaim: Explainable Neural Claim Verification Using Rationalization

Gurrapu,

Huang,

Batarseh

2023

Preprint

View full text Add to dashboard Cite

With the advent of deep learning, text generation language models have improved dramatically, with text at a similar level as human-written text. This can lead to rampant misinformation because content can now be created cheaply and distributed quickly. Automated claim verification methods exist to validate claims, but they lack foundational data and often use mainstream news as evidence sources that are strongly biased towards a specific agenda. Current claim verification methods use deep neural network models and complex algorithms for a high classification accuracy but it is at the expense of model explainability. The models are black-boxes and their decisionmaking process and the steps it took to arrive at a final prediction are obfuscated from the user. We introduce a novel claim verification approach, namely: ExClaim, that attempts to provide an explainable claim verification system with foundational evidence. Inspired by the legal system, ExClaim leverages rationalization to provide a verdict for the claim and justifies the verdict through a natural language explanation (rationale) to describe the model's decision-making process. ExClaim treats the verdict classification task as a question-answer problem and achieves a performance of 0.93 F1 score. It provides subtasks explanations to also justify the intermediate outcomes. Statistical and Explainable AI (XAI) evaluations are conducted to ensure valid and trustworthy outcomes. Ensuring claim verification systems are assured, rational, and explainable is an essential step toward improving Human-AI trust and the accessibility of black-box systems.

show abstract

Pretrained Language Models for Text Generation: A Survey

Cited by 21 publications

References 150 publications

MVP: Multi-task Supervised Pre-training for Natural Language Generation

MVP: Multi-task Supervised Pre-training for Natural Language Generation

Learning to Diversify for Product Question Generation

ExClaim: Explainable Neural Claim Verification Using Rationalization

Contact Info

Product

Resources

About