Evaluating prose style transfer with the Bible

Abstract

In the prose style transfer task a system, provided with text input and a target prose style, produces output which preserves the meaning of the input text but alters the style. These systems require parallel data for evaluation of results and usually make use of parallel data for training. Currently, there are few publicly available corpora for this task. In this work, we identify a high-quality source of aligned, stylistically distinct text in different versions of the Bible. We provide a standardized split, into training, development and testing data, of the public domain versions in our corpus. This corpus is highly parallel since many Bible versions are included. Sentences are aligned due to the presence of chapter and verse numbers within all versions of the text. In addition to the corpus, we present the results, as measured by the BLEU and PINC metrics, of several models trained on our data which can serve as baselines for future research. While we present these data as a style transfer corpus, we believe that it is of unmatched quality and may be useful for other natural language tasks as well.

An Empirical Study on Multi-Task Learning for Text Style Transfer and Paraphrase Generation

“…We treat these corpora as domain-unspecific and stylistically neutral. As sources, we used the paraphrases corpus presented in Wieting and Gimpel (2018), MultiNLI corpus (Williams et al, 2018), Quora Kaggle dataset (Quora, 2017) and Bible corpus (Carlson et al, 2018 In the M2A task we removed the Bible subcorpus from the dataset of paraphrases to perform the multi-task training with a small amount of style data (like in the N2C task).…”

“…The idea of using the Bible as a parallel corpus suitable for ST tasks was presented in Carlson et al (2018). The authors claim that traditionally used sentence and verse demarcation makes for easy sentence alignment.…”

“…Style transfer is extensively studied in academic papers (Li et al, 2018a;Rao and Tetreault, 2018;Carlson et al, 2018;Jhamtani et al, 2017) and also gains popularity in commercialized chatbots. Amazon Alexa introduced styles mimicking celebrities that replace the original Alexa voice (Amazon, 2020).…”

“…This is a convenient approach, because we have publicly available large parallel corpora for paraphrasing, but rather small parallel corpora of style transformations we want to achieve. We perform our experiments using openly available paraphrase sources (Rao and Tetreault, 2018;Carlson et al, 2018;Quora, 2017;Wieting and Gimpel, 2018;Williams et al, 2018) and two ST corpora, one for each task: the internal "cute person" style corpus and the processed Bible texts corpus composed of publicly available sources. We examine two style transformations: neutral-to-cute and modern-to-antique (the latter one on different Bible translations).…”

Decomposing Textual Information For Style Transfer

“…Moreover, the system that does not rely on an idea of disentangled latent representations at all shows weaker results than systems with high information disentanglement. It is important to note that there is a variety of methods to assess the quality of style transfer such as PINC (Paraphrase In N-gram Changes) score (Carlson et al, 2018), POS distance (Tian et al, 2018), language fluency , etc. The methodology of style transfer quality assessment is addressed in detail in (Tikhonov et al, 2019), but BLEU between output and input is a very natural all-purpose metric for the task of such type that is common in the style transfer literature.…”