A Standardized Project Gutenberg Corpus for Statistical Analysis of Natural Language and Quantitative Linguistics

Abstract

The use of Project Gutenberg (PG) as a text corpus has been extremely popular in statistical analysis of language for more than 25 years. However, in contrast to other major linguistic datasets of similar importance, no consensual full version of PG exists to date. In fact, most PG studies so far either consider only a small number of manually selected books, leading to potential biased subsets, or employ vastly different pre-processing strategies (often specified in insufficient details), raising concerns regarding the reproducibility of published results. In order to address these shortcomings, here we present the Standardized Project Gutenberg Corpus (SPGC), an open science approach to a curated version of the complete PG data containing more than 50,000 books and more than 3 × 10 9 word-tokens. Using different sources of annotated metadata, we not only provide a broad characterization of the content of PG, but also show different examples highlighting the potential of SPGC for investigating language variability across time, subjects, and authors. We publish our methodology in detail, the code to download and process the data, as well as the obtained corpus itself on 3 different levels of granularity (raw text, timeseries of word tokens, and counts of words). In this way, we provide a reproducible, pre-processed, full-size version of Project Gutenberg as a new scientific resource for corpus linguistics, natural language processing, and information retrieval.

Pull out all the stops: Textual analysis via punctuation sequences

“…We use the API functionality of Project Gutenberg [17] to obtain our document corpus and the natural-language-processing (NLP) library spaCy [20] to extract an ordered punctuation sequence from each document. Using data from Project Gutenberg requires various filtering and cleaning steps before it is possible to meaningfully perform statistical analysis [14]. We describe our steps below.…”

“…In some our computational experiments, we use the following metadata: author birth year, author death year, and document "bookshelf" (which we term document "genre", as that is what it appears to represent). The authors of [14] pointed out recently that "bookshelf" may be better suited than "subject" to practical purposes such as text classification, because the former constitute broader categories and provide a unique assignment of labels to documents.…”

“…To our knowledge, most attempts thus far at author recognition of literary documents have used data sets that are of significantly smaller scale than our corpus [14,35]. One recent example of author analysis from a corpus extracted from Project Gutenberg is the one in Qian et al [39].…”

“…For our study, we use Project Gutenberg [17] to obtain a large corpus of documents, and we extract an ordered set of punctuation marks from each document in the corpus (see section 2). Although Project Gutenberg is a popular database for the statistical analysis of language, most previous studies that have used it have considered only a small number of manually selected documents [14]. Broadly, our goal is to investigate the following question: Do punctuation marks encode stylistic information about an author, a genre, or a time period?…”

“…To assess whether our observations extend beyond our Gutenberg corpus, it is necessary to conduct further experiments (e.g., on a larger corpus, across different e-book sources, and so on). For example, it is desirable to repeat our analysis using the "Text data" level of granularity in the recently introduced (while we were in the final stages of writing this article) Standardized Project Gutenberg Corpus [14].…”