Analysing Lexical Semantic Change with Contextualised Word Representations

Abstract

This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method captures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction.

ELMo and BERT in Semantic Change Detection for Russian

“…This provides new opportunities for diachronic analysis: for example, it is possible to group similar token representations and measure a diversity of such representations, while predefined number of senses is not strictly necessary. Thus, currently there is an increased interest in the topic of language change detection using contextualized word embeddings [9,10,14,21,27,28].…”

“…Evolution of each word was measured by comparing distributions of senses in different time slices. In [9] and [10], a pre-trained BERT model was used to obtain representations of word usages in an unsupervised fashion, without predefined list or number of senses. Representations with similar usages then were clustered using k-Means algorithm and distributions of word's usages in these clusters were used in two metrics for quantifying the degree of semantic change: entropy difference and Jensen-Shannon divergence.…”

“…[28] tested Affinity Propagation algorithm for usage clusterization and showed that it is consistently better than k-Means. Finally, [21] applied approaches similar to [10], but also analyzing ELMo models and adding cosine similarity of average vectors as a measure. Their algorithms were evaluated on Subtask 2 (ranking) of SemEval-2020 Task 1 [37] for four different languages and strongly outperformed the baselines.…”

“…We need to compare the resulting clusters of word's usages in two time periods. We used the method proposed in [9]. Namely, we clustered all embeddings of each word w from the two time periods (u t 1 , u t 2 , ..., u t n ) and (u t+1 1 , u t+1 2 , ..., u t+1 m ).…”

Will it Unblend?

“…This criterion can be viewed as a form of linguistic generalization, and if satisfied, enables downstream models to produce consistent results across related words. To test this criterion, we compute the vector similarities between the contextualized representations of complex words and their components, a method that coheres with human judgments of contextual semantic similarity (Giulianelli et al, 2020). We probe BERT 11 with synthetic inputs constructed by replacing each complex word with its spacedelimited bases.…”