Recent advancements in self-attention neural network architectures have raised the bar for openended text generation. Yet, while current methods are capable of producing a coherent text which is several hundred words long, attaining control over the content that is being generated-as well as evaluating it-are still open questions. We propose a controlled generation task which is based on expanding a sequence of facts, expressed in natural language, into a longer narrative. We introduce human-based evaluation metrics for this task, as well as a method for deriving a large training dataset. We evaluate three methods on this task, based on fine-tuning pre-trained models. We show that while auto-regressive, unidirectional Language Models such as GPT2 produce better fluency, they struggle to adhere to the requested facts. We propose a plan-andcloze model (using fine-tuned XLNet) which produces competitive fluency while adhering to the requested content.

Recent advancements in self-attention neural network architectures have raised the bar for openended text generation. Yet, while current methods are capable of producing a coherent text which is several hundred words long, attaining control over the content that is being generated-as well as evaluating it-are still open questions. We propose a controlled generation task which is based on expanding a sequence of facts, expressed in natural language, into a longer narrative. We introduce human-based evaluation metrics for this task, as well as a method for deriving a large training dataset. We evaluate three methods on this task, based on fine-tuning pre-trained models. We show that while auto-regressive, unidirectional Language Models such as GPT2 produce better fluency, they struggle to adhere to the requested facts. We propose a plan-andcloze model (using fine-tuned XLNet) which produces competitive fluency while adhering to the requested content.This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/.1 We use the term 'fact' as used in Open Information Extraction: a factive sentence in the text, which may or may not correspond to an actual fact in the real world. FactsFact 1: Italian immigrant Antonio Tony Camonte kills Big Louis Costillo Fact 2: Johnny takes control of South Side with Tony as key lieutenant Fact 3: Tony starts ignoring orders shooting up bars belonging to O'Hara and attracting attention of police and rival gangsters Fact 4: police move to arrest Tony for Guino 's murder Fact 5: Cesca ends up helping him to fight police Corresponding textIn 1920s Chicago, Italian immigrant Antonio "Tony" Camonte acts on the orders of Italian mafioso John "Johnny" Lovo and kills "Big" Louis Costillo, the leading crime boss of the city's South Side. Johnny then takes control of the South Side with Tony as his key lieutenant, selling large amounts of illegal beer to speakeasies and muscling in on bars run by rival outfits. However, Johnny repeatedly warns Tony not to mess with the Irish gangs led by O'Hara, who runs the North Side. Tony soon starts ignoring these orders, shooting up bars belonging to O'Hara, and attracting the attention of the police and rival gangsters. Johnny realizes that Tony is out of control and has ambitions to take his position. Meanwhile, Tony pursues Johnny's girlfriend Poppy with increasing confidence. At first, she is dismissive of him but pays him more attention as his reputation rises. At one point, she visits his "gaudy" apartment where he shows her his view of an electric billboard advertising Cook's Tours, which features the slogan that has inspired him: "The World is Yours." Tony eventually decides to declare war and take over the North Side. He sends the coin flipping Guino Rinaldo, one of his best men and also his close friend, to kill O'Hara in a florist's shop that he uses as his base. This brings heavy retaliation from the North Side gangs, now led by Gaffney and armed with Thompson s...