Unsupervised Hierarchical Story Infilling

Ippolito, Daphne; Grangier, David; Callison-Burch, Chris; Eck, Douglas

doi:10.18653/v1/w19-2405

Cited by 38 publications

(50 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Story generation. Recent work seeks to generate stories given a title and storyline (Yao et al, 2019), entities (Clark et al, 2018), premise (Fan et al, 2018), or surrounding context and rare words (Ippolito et al, 2019). Our work differs in that we aim to build systems capable of making predictions based only on text context, rather than aspects specific to stories (e.g.…”

Section: Related Workmentioning

confidence: 99%

Enabling Language Models to Fill in the Blanks

Donahue¹,

Lee²,

Liang³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

120

127

View full text Add to dashboard Cite

We present a simple approach for text infilling, the task of predicting missing spans of text at any position in a document. While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling-a special case of infilling where text is predicted at the end of a document. In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine-tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling, can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machinegenerated in the domain of short stories.

show abstract

Section: Related Workmentioning

confidence: 99%

Enabling Language Models to Fill in the Blanks

Donahue¹,

Lee²,

Liang³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

120

127

View full text Add to dashboard Cite

show abstract

“…proposes an iterative inference algorithm based on gradient search for text infilling. For story infilling, (Ippolito et al, 2019) first predicts rare words in the missing span, and then generates text conditioned on these words. SpanBERT (Joshi et al, 2020) masks random contiguous spans and (pre-)trains a language model to predict tokens in the span.…”

Section: Related Workmentioning

confidence: 99%

“…Generating a span of missing tokens in a text chunk, known as "text infilling," has attracted many attentions recently (Zhu et al, 2019;Song et al, 2019;Ippolito et al, 2019;Joshi et al, 2020). Here we study the related but somewhat different task of "sentence infilling."…”

Section: Introductionmentioning

confidence: 99%

INSET: Sentence Infilling with INter-SEntential Transformer

Huang

Zhang

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Missing sentence generation (or sentence infilling) fosters a wide range of applications in natural language generation, such as document auto-completion and meeting note expansion. This task asks the model to generate intermediate missing sentences that can syntactically and semantically bridge the surrounding context. Solving the sentence infilling task requires techniques in natural language processing ranging from understanding to discourselevel planning to generation. In this paper, we propose a framework to decouple the challenge and address these three aspects respectively, leveraging the power of existing largescale pre-trained models such as BERT and GPT-2. We empirically demonstrate the effectiveness of our model in learning a sentence representation for generation and further generating a missing sentence that fits the context.

show abstract

“…There has been a variety of work focusing on generating stories in plot-controllable, plan-driven, or constrained ways (e.g. (Riedl and Young, 2010;Fan et al, 2018;Peng et al, 2018;Jain et al, 2017;Lebowitz, 1987;Ippolito et al, 2019;Pérez y Pérez and Sharples, 2001)). Similar work in creative generation has conditioned on keywords for poetry generation (Yan, 2016;Ghazvininejad et al, 2016;Wang et al, 2016).…”

Section: Controllable Story Generationmentioning

confidence: 99%

PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking

Rashkin¹,

Çelikyılmaz²,

Choi³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

We propose the task of outline-conditioned story generation: given an outline as a set of phrases that describe key characters and events to appear in a story, the task is to generate a coherent narrative that is consistent with the provided outline. This task is challenging as the input only provides a rough sketch of the plot, and thus, models need to generate a story by interweaving the key points provided in the outline. This requires the model to keep track of the dynamic states of the latent plot, conditioning on the input outline while generating the full story. We present PLOTMACHINES, a neural narrative model that learns to transform an outline into a coherent story by tracking the dynamic plot states. In addition, we enrich PLOTMACHINES with high-level discourse structure so that the model can learn different writing styles corresponding to different parts of the narrative. Comprehensive experiments over three fiction and non-fiction datasets demonstrate that large-scale language models, such as GPT-2 and GROVER, despite their impressive generation performance, are not sufficient in generating coherent narratives for the given outline, and dynamic plot state tracking is important for composing narratives with tighter, more consistent plots.

show abstract

Unsupervised Hierarchical Story Infilling

Cited by 38 publications

References 20 publications

Enabling Language Models to Fill in the Blanks

Enabling Language Models to Fill in the Blanks

INSET: Sentence Infilling with INter-SEntential Transformer

PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking

Contact Info

Product

Resources

About