Evaluation of text-level measures of lexical dispersion: Robustness and consistency

Abstract

The traditional approach to measuring lexical dispersion is to form corpus parts of equal size and then compare the occurrence rate of an item across these units. In recent methodological work, this strategy has met with criticism due to its ignorance to corpus structure. Dispersion, it is argued, should be measured across linguistically meaningful units such as the individual text files constituting the corpus. Though desirable on linguistic grounds, a shift to texts as the unit of analysis raises new methodological issues. While the ability of dispersion measures to handle unevenly-sized corpus units has received attention in the literature, the question of how existing metrics perform in these novel settings has only been partly addressed. This paper aims to shed light on relevant statistical properties of a wide range of text-level dispersion measures. Specifically, we consider the robustness of different indicators, i.e. whether they are (overly) sensitive to data situations that can arise when texts differ (considerably) in length. We use hypothetical data scenarios to identify weak spots in existing measures, and then propose modifications to DP- and DA-related indexes to implement useful statistical properties and effect more resistant estimators. Along with the other measures, these are then evaluated against actual corpus data drawn from the BNC. We observe that adapted DP- and DA-variants perform at least as well as their original versions. Our permutation-based simulation study also demonstrates that Carroll’s D2 shows the same weakness as Juilland’s D, i.e. a noticeable sensitivity to the number of units that enter the analysis.

Count regression models for keyness analysis

“…In corpus B, instances are more densely clustered, and there are large stretches of text where the item does not occur. In the corpus-linguistic sense, then, the dispersion of the item is higher in corpus A (see Gries 2008Gries , 2020Sönning 2022b). This dot marks how often the item appeared in the text.…”

“…As Table 1 shows, these keyness dimensions allow us to form four linguistically meaningful classes of metrics. For reasons of space, we cannot provide details about the individual measures here, and we refer the reader to Gabrielatos (2018), Rayson & Potts (2020), Gries (2020), and Sönning (2022a, 2022b. The four-way arrangement in Table 1 offers a constructive point of departure for keyness analysis, since it requires the analyst to first consider which features of keyness to emphasize when looking for typical items in the target corpus.…”

Evaluation of keyness metrics: Reliability and interpretability

“…texts) of different length (cf. Gries 2020; Sönning 2022a). An overview of different dispersion measures is provided in Gries (2020) and Sönning (2022a).…”

“…Gries 2020; Sönning 2022a). An overview of different dispersion measures is provided in Gries (2020) and Sönning (2022a). For an assessment of the generality of an item, we will consider the following measures: D (Juilland et al 1970), D2 (Carroll 1970), Sadj (Rosengren 1972), DP (Gries 2008;Lijffit & Gries 2012;, DA (Wilcox 1973;, and DKL (Gries 2020(Gries , 2021.…”