Literary Intertextual Semantic Change Detection: Application and Motivation for Evaluating Models on Small Corpora

Abstract

Lexical semantic change detection is the study of how words change meaning between corpora. While Schlechtweg et al. (2020) standardized datasets and evaluation metrics for this shared task, for those interested in applying semantic change detection models to small corpora-e.g., in the digital humanities-there is a need for evaluation involving much smaller datasets. We present a method and open-source code pipeline for downsampling the SemEval-2020 Task 1 corpora while preserving gold standard measures of semantic change. We then evaluate several high-performing unsupervised models on these downsampled corpora, and find that the models experience both dramatically decreased performance (average 67% decrease) and high variance. Finally, we propose a novel application to the digital humanities: literary intertextual semantic change detection, the production of a ranked list of words by degree of semantic change between two books. We then provide a case study of this application to Fanon's The Wretched of the Earth and Hartman's Scenes of Subjection and find that semantic change detection models-even with their current limited performance on small corpora-may still produce fruitful avenues of exploration for literary scholars.