Conceptual Change and Distributional Semantic Models: an Exploratory Study on Pitfalls and Possibilities

Abstract

Studying conceptual change using embedding models has become increasingly popular in the Digital Humanities community, while critical observations about them have received less attention. This paper investigates what the impact of known pitfalls can be on the conclusions drawn in a digital humanities study through the use case of "Racism" in the 20th century. In addition, we suggest an approach for modeling a complex concept in terms of words and relations representative of the conceptual system. Our results show that different models created from the same data yield different results, but also indicate that (i) using different model architectures, (ii) comparing different corpora and (iii) comparing to control words and relations can help to identify which results are solid and which may be due to artefacts. We propose guidelines to conduct similar studies, but also note that more work is needed to fully understand how we can distinguish artefacts from actual conceptual changes.

A Survey of Race, Racism, and Anti-Racism in NLP

“…Models Some papers have found evidence that model instances or architectures can change the racial biases of outputs produced by the model. Sommerauer and Fokkens (2019) find that the word embedding associations around words like 'race' and 'racial' change not only depending on the model architecture used to train embeddings, but also on the specific model instance used to extract them, perhaps because of differing random seeds. Kiritchenko and Mohammad (2018) examine gender and race biases in 200 sentiment analysis systems submitted to a shared task and find different levels of bias in different systems.…”

Expert Concept-Modeling Ground Truth Construction for Word Embeddings Evaluation in Concept-Focused Domains

“…In stressing the danger posed by using little understood DSMs methodologies to detect concept change, Sommerauer and Fokkens (2019) fix an initial articulation of the concept of racism and a number of target terms as proxies for the subconcepts to be detected in the COHA and the English Google n-gram corpus; they also offer guidelines for sound methodology. Our work can be seen as thoroughly expanding on the first guideline, according to which a "wide range of verifiable hypotheses" need to be offered "to study the overall question before diving into actual changes".…”

“…In certain specialised textual domains in which fine-grained study of concepts is traditionally performed exclusively by humans, the use of word embeddings on corpora of a few million tokens -too big a size for manual processing in a reasonable time -is, potentially, a game-changer (Betti et al, 2019). Technology needs validation before it can be game-changing, however (Hellrich, 2019;Sommerauer and Fokkens, 2019): word embedding techniques cannot gain a foothold in traditional domains such as history or philosophy if they cannot reasonably approximate or reliably support expert work, or if they introduce unknown biases (Fokkens et al, 2014). Unfortunately, the performance of DSMs is notoriously difficult to evaluate; it is even controversial how one should carry out evaluations in generic, big corpora domains (Gladkova and Drozd, 2016;Bakarov, 2018).…”

The challenges of temporal alignment on Twitter during crises

“…Patterns of language usage change over time, often in predictable and analyzable ways (Hamilton et al, 2016b;Kulkarni et al, 2015;Sommerauer and Fokkens, 2019). As language changes, the applicability of NLP systems can be negatively impacted.…”