Can Machines Tell Stories? A Comparative Study of Deep Neural Language Models and Metrics

Abstract

Massive textual content has enabled rapid advances in natural language modeling. The use of pre-trained deep neural language models has significantly improved natural language understanding tasks. However, the extent to which these systems can be applied to content generation is unclear. While a few informal studies have claimed that these models can generate 'high quality' readable content, there is no prior study on analyzing the generated content from these models based on sampling and fine-tuning hyperparameters. We conduct an in-depth comparison of several language models for open-ended story generation from given prompts. Using a diverse set of automated metrics, we compare the performance of transformer-based generative models -OpenAI's GPT2 (pre-trained and fine-tuned) and Google's pretrained Transformer-XL and XLNet to human-written textual references. Studying inter-metric correlation along with metric ranking reveals interesting insights -the high correlation between the readability scores and word usage in the text. A study of the statistical significance and empirical evaluations between the scores (human and machine-generated) at higher sampling hyperparameter combinations (t = {0.75, 1.0}, k = {100, 150, 250}) reveal that the top pre-trained and fine-tuned models generated samples condition well on the prompt with an increased occurrence of unique and difficult words. The GPT2-medium model fine-tuned on the 1024 Byte-pair Encoding (BPE) tokenized version of the dataset along with pre-trained Transformer-XL models generated samples close to human written content on three metrics: prompt-based overlap, coherence, and variation in sentence length. A study of overall model stability and performance shows that fine-tuned GPT2 language models have the least deviation in metric scores from human performance.

Evaluating the machine learning models based on natural language processing tasks

“…The metric score of the content generated by the models was compared with the metric score of human written content; no model was close to human written content. However, the fine-tuned GPT-2 models and the transformer-extra long (XL) models were the ones that had lowest deviation in the metrics whereas the XL Net models had a high value of deviation [10]. Data imbalance is one of the major problems in classification, lack of a dataset belonging to any of the class labels could result in data imbalance and further lead to poor classification performance results.…”

KDDT: Knowledge Distillation-Empowered Digital Twin for Anomaly Detection

“…Perplexity of LM. Perplexity is commonly used to evaluate probabilistic models, particularly LMs [7]. In the context of an LM, perplexity measures the average likelihood of its predictions for the next token.…”

Efficiency of Automatic Text Generators for Online Review Content Generation

“…As all these models are pretrained and publicly available, they represent a significant threat in the field of misleading reviews, as anyone can retrain the model on a specific field and use them for malicious purposes. GPT2 has been used extensively for text generation (Das and Verma, 2020) and review generation (Salminen et al, 2022). BERT (Bidirectional Encoding Representations of Transformers) is another transformer-based model that has been frequently utilized for language interpretation tasks, although some recent works have employed this model for text generation (Devlin et al, 2019).…”