Significance testing of word frequencies in corpora

Abstract

Finding out whether a word occurs significantly more often in one text or corpus than in another is an important question in analysing corpora. As noted by Kilgarriff (2005), the use of the  2 and log-likelihood ratio tests is problematic in this context, as they are based on the assumption that all samples are statistically independent of each other. However, words within a text are not independent. As pointed out in Kilgarriff (2001) and Paquot & Bestgen (2009), it is possible to represent the data differently and employ other tests, such that we assume independence at the level of texts rather than individual words. This allows us to account for the distribution of words within a corpus. In this article we compare the significance estimates of various statistical tests in a controlled resampling experiment and in a practical setting, studying differences between texts produced by male and female fiction writers in the British National Corpus. We find that the choice of the test, and hence data representation, matters. We conclude that significance testing can be used to find consequential differences between corpora, but that assuming independence between all words may lead to overestimating the significance of the observed differences, especially for poorly dispersed words. We recommend the use of the t-test, Wilcoxon rank-sum test, or bootstrap test for comparing word frequencies across corpora.

Getting rid of the Chi-square and Log-likelihood tests for analysing vocabulary differences between corpora

“…However, these tests, according to the way they are used to analyse lexical differences between corpora, are inadequate, as has already been pointed out by several authors, and should no longer be used (Bestgen, 2012(Bestgen, , 2014Brezina & Meyerhoff, 2014;Kilgarriff, 1996Kilgarriff, , 2005Lijffijt, Nevalainen, Säily, Papapetrou, Puolamäki & Mannila, 2016). The aim of this paper is to help researchers to abandon them by explaining in detail the problem they pose and its origin, by showing why several possible solutions are ineffective and by recommending two valid and efficient statistical tests.…”

Understanding Accessibility as a Process through the Analysis of Feedback from Disabled Students

“…There are several approaches to define the 'keyness' of words in a corpus of text [13]. Here, we use Rayson's (2008) approach to calculate the log-likelihood ratio between the frequency of a word from one dataset compared to the frequency of that word in a reference dataset.…”

Approximating Style by N-gram-based Annotation

“…To achieve this, we use the t-test as suggested by Paquot and Bestgen (2009) and Lijffijt et al (2014). One of the benefits of the t-test is that it takes variation within the corpora into account.…”