Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View

Abstract

Diachronic word embeddings have been widely used in detecting temporal changes. However, existing methods face the meaning conflation deficiency by representing a word as a single vector at each time period. To address this issue, this paper proposes a sense representation and tracking framework based on deep contextualized embeddings, aiming at answering not only what and when, but also how the word meaning changes. The experiments show that our framework is effective in representing fine-grained word senses, and it brings a significant improvement in word change detection task. Furthermore, we model the word change from an ecological viewpoint, and sketch two interesting sense behaviors in the process of language evolution, i.e. sense competition and sense cooperation.

Sequential Modelling of the Evolution of Word Representations for Semantic Change Detection

“…Future work can use anomaly detection approaches operating on our model's predicted word vectors to detect anomalies in a word's representation across time. We also plan to investigate different architectures, such as Variational Autoencoders (Kingma and Welling, 2014), and incorporate contextual representations (Devlin et al, 2019;Hu et al, 2019) to detect new senses of words. A limitation of our work is that it has been tested on a single dataset, where 65 words have undergone semantic change; testing our models in datasets of different duration and in different languages will provide clearer evidence of their effectiveness.…”

LessLex: Linking Multilingual Embeddings to SenSe Representations of LEXical Items

“…Further investigations are possible, stemming from the fact that senses and terms share the same multilingual semantic space: For example, we are allowed to compare and unveil meaning connections between terms across different languages. Such capabilities can be useful in characterizing subtle and elusive meaning shift phenomena, such as diachronic sense modeling (Hu, Li, and Liang 2019) and conceptual misalignment, which is a well-known issue, for example, in the context of automatic translation. This issue has been approached, for the translation of European laws, through the design of formal ontologies (Ajani et al 2010).…”

ELMo and BERT in Semantic Change Detection for Russian

“…This provides new opportunities for diachronic analysis: for example, it is possible to group similar token representations and measure a diversity of such representations, while predefined number of senses is not strictly necessary. Thus, currently there is an increased interest in the topic of language change detection using contextualized word embeddings [9,10,14,21,27,28].…”