A Plug-and-Play Method for Controlled Text Generation

Pascual, Damián; Egressy, Béni; Meister, Clara; Cotterell, Ryan; Wattenhofer, Roger

doi:10.18653/v1/2021.findings-emnlp.334

Cited by 26 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, PPCM (Madotto et al 2020) updates the hidden state in direction of the attribute enhancement to generate attribute-aware conversations. Pascual (Pascual et al 2021) designed a complex plug-and-play architecture to ensure that the generated content contains specific keywords. GeDi (Krause et al 2021) and its extension (Lin and Riedl 2021) can accelerate the decoding process of the PPLM, but they assume that the model is trained using large-scale labeled datasets, which are unrealizable for text infilling.…”

Section: Constrained Text Generationmentioning

confidence: 99%

Plug-and-Play Attribute-Aware Text Infilling via A New Attention Mechanism and Two-Level Positional Encoding

Li,

Funakoshi,

Okumura

2023

Journal of Natural Language Processing

View full text Add to dashboard Cite

Text infilling aims to restore incomplete texts by filling in blanks and has attracted increasing attention recently because of its wide application in ancient text restoration, conversation generation, and text rewriting. However, attribute-aware text infilling is yet to be explored, and existing methods seldom focus on the infilling length of each blank or the number and location of the blanks. In this study, we propose a plug-and-play Attribute-aware Text Infilling method using a Pre-trained language model (A-TIP) that contains a text-infilling component and a plug-and-play discriminator. Specifically, we first designed a unified text-infilling component with modified attention mechanisms and intra-and inter-blank positional encoding to better perceive the number of blanks and the infilling length for each blank. We then propose a plug-and-play discriminator to guide generation and improve attribute relevance without decreasing text fluency. Finally, automatic and human evaluations on three open-source datasets indicate that, compared to all the baselines, A-TIP achieves state-of-the-art performance. An additional ablation study demonstrated the robustness of A-TIP.

show abstract

Section: Constrained Text Generationmentioning

confidence: 99%

Plug-and-Play Attribute-Aware Text Infilling via A New Attention Mechanism and Two-Level Positional Encoding

Li,

Funakoshi,

Okumura

2023

Journal of Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…A representative of this group of methods is PPLM [Dathathri et al, 2020], which first trains an attribute discriminant model and then uses it to guide language model to generate the text with corresponding topic or sentiment. This group also includes the Keyword2Text method [15], which can be applied to an existing autoregressive language model without additional training. The idea of the method is to shift the output distribution of the language generation model to the semantic space of a given guide word in the word2vec or GloVe vector space.…”

Section: Previous Workmentioning

confidence: 99%

MaxProb: Controllable Story Generation from Storyline

Vychegzhanin,

Kotelnikova,

Sergeev

et al. 2023

Computational Linguistics and Intellectual Technologies”

View full text Add to dashboard Cite

Controllable story generation towards keywords or key phrases is one of the purposes of using language models. Recent work has shown that various decoding strategies prove to be effective in achieving a high level of language control. Such strategies require less computational resources compared to approaches based on fine-tuning pre-trained language models. The paper proposes and investigates the method MaxProb of controllable story generation in Russian, which works at the decoding stage in the process of text generation. The method uses a generative language model to estimate the probability of its tokens in order to shift the content of the text towards the guide phrase. The idea of the method is to generate a set of different small sequences of tokens from the language model vocabulary, estimate the probability of following the guide phrase after each sequence, and choose the most probable sequence. The method allows evaluating the consistency of the token sequence for the transition from the prompt to the guide phrase. The study was carried out using the Russian-language corpus of stories with extracted events that make up the plot of the story. Experiments have shown the effectiveness of the proposed method for automatically creating stories from a set of plot phrases.

show abstract

“…A representative controllable PLM is the Plug and Play Language Model [31], also called PPLM, which combines a PLM with one or more simple attribute classifiers that guide text generation without any further training of the PLM. Several studies achieved the goal of controllablility from a distributional view [89,140]. For example, Pascual et al [140] present a plug-and-play decoding method, which can be described in a single sentence: given a topic or keyword, the model add a shift to the probability distribution over the vocabulary towards semantically similar words.…”

Section: Optimization Viewmentioning

confidence: 99%

“…Several studies achieved the goal of controllablility from a distributional view [89,140]. For example, Pascual et al [140] present a plug-and-play decoding method, which can be described in a single sentence: given a topic or keyword, the model add a shift to the probability distribution over the vocabulary towards semantically similar words.…”

Section: Optimization Viewmentioning

confidence: 99%

Pretrained Language Models for Text Generation: A Survey

Li¹,

Tang²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Text Generation aims to produce plausible and readable text in human language from input data. The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). Grounding text generation on PLMs is seen as a promising direction in both academia and industry. In this survey, we present the recent advances achieved in the topic of PLMs for text generation. In detail, we begin with introducing three key points of applying PLMs to text generation: 1) how to encode the input data as representations preserving input semantics which can be fused into PLMs; 2) how to design a universal and performant architecture of PLMs served as generation models; and 3) how to optimize PLMs given the reference text and ensure the generated text satisfying special text properties. Then, we figure out several challenges and future directions within each key point. Next, we present a summary of various useful resources and typical text generation applications to work with PLMs. Finally, we conclude and summarize the contribution of this survey.CCS Concepts: • Computing methodologies → Natural language generation.

show abstract

A Plug-and-Play Method for Controlled Text Generation

Cited by 26 publications

References 20 publications

Plug-and-Play Attribute-Aware Text Infilling via A New Attention Mechanism and Two-Level Positional Encoding

Plug-and-Play Attribute-Aware Text Infilling via A New Attention Mechanism and Two-Level Positional Encoding

MaxProb: Controllable Story Generation from Storyline

Pretrained Language Models for Text Generation: A Survey

Contact Info

Product

Resources

About