Changing the Mind of Transformers for Topically-Controllable Language Generation

Abstract

Large Transformer-based language models can aid human authors by suggesting plausible continuations of text written so far. However, current interactive writing assistants do not allow authors to guide text generation in desired topical directions. To address this limitation, we design a framework that displays multiple candidate upcoming topics, of which a user can select a subset to guide the generation. Our framework consists of two components: (1) a method that produces a set of candidate topics by predicting the centers of word clusters in the possible continuations, and ( 2) a text generation model whose output adheres to the chosen topics. The training of both components is self-supervised, using only unlabeled text. Our experiments demonstrate that our topic options are better than those of standard clustering approaches, and our framework often generates fluent sentences related to the chosen topics, as judged by automated metrics and crowdsourced workers.

Harnessing the power of LLMs: Evaluating human-AI text co-creation through the lens of news headline generation

“…Guiding model output entails human intervention in the automatic text generation process, such as specifying crucial parts of input text to be included in the output (Gehrmann et al, 2019), or providing semantic prompts or style parameters (Osone et al, 2021;Chang et al, 2021;Strobelt et al, 2022). Our study involves human guidance in headline generation by specifying keyword prompts.…”

Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications

“…Paul, Chang, and McCallum (2021) use the co-occurring relation between a sentence pattern and its entity pair to improve relation extraction in Verga et al (2016). Chang et al (2021) use the co-occurring relation between a context paragraph and its subsequent words to control the topics of language generation. In the future, the approach might also be used to improve the efficiency of document similarity estimation (Luan et al 2020).…”