Training Temporal Word Embeddings with a Compass

Abstract

Temporal word embeddings have been proposed to support the analysis of word meaning shifts during time and to study the evolution of languages. Different approaches have been proposed to generate vector representations of words that embed their meaning during a specific time interval. However, the training process used in these approaches is complex, may be inefficient or it may require large text corpora. As a consequence, these approaches may be difficult to apply in resource-scarce domains or by scientists with limited in-depth knowledge of embedding models. In this paper, we propose a new heuristic to train temporal word embeddings based on the Word2vec model. The heuristic consists in using atemporal vectors as a reference, i.e., as a compass, when training the representations specific to a given time interval. The use of the compass simplifies the training process and makes it more efficient. Experiments conducted using stateof-the-art datasets and methodologies suggest that our approach outperforms or equals comparable approaches while being more robust in terms of the required corpus size.

Words with Consistent Diachronic Usage Patterns are Learned Earlier: A Computational Analysis Using Temporally Aligned Word Embeddings

“…Over the past few years, different methods have been proposed to align different temporal slices of the same corpus with the goal of quantifying lexical semantic shift in a data-driven and scalable way (Bianchi et al, 2020;Di Carlo et al, 2019;Dubossarsky et al, 2019;Hamilton et al, 2016b;Kim et al, 2014;Kulkarni et al, 2015;Rudolph & Blei, 2018;Tahmasebi et al, 2018;Yao et al, 2018). For example, Hamilton et al (2016b) used Procrustes transformation to align embeddings, while Yao et al (2018) used a joint optimization procedure.…”

“…For example, Hamilton et al (2016b) used Procrustes transformation to align embeddings, while Yao et al (2018) used a joint optimization procedure. More recently, Di Carlo et al (2019) proposed the temporally aligned word embeddings with a compass (TWEC 7 ) model, which extends the continuous bag of words (CBoW) architecture (Mikolov et al, 2013). The CBoW architecture is a neural network with one hidden layer and uses two matrices to learn lexical representations, a target matrix and a context matrix.…”

“…We then use several measures derived using such method to analyze the relation between language acquisition and language evolution. While previous studies addressing this relation (Monaghan, 2014; Monaghan & Roberts, 2019; Vejdemo & Hörberg, 2016) relied on small sample sizes and expert annotations, limiting the scope of their results, we leverage recent advances in natural language processing (NLP) to track language behavior over time and learn linguistic representations in an unsupervised and scalable way (Bianchi, Di Carlo, Nicoli, & Palmonari, 2020; Di Carlo, Bianchi, & Palmonari, 2019).…”

“…While the four aforementioned limitations are distinct and could be addressed orthogonally, we propose to address them organically. Rather than using curated resources, we exploit recent NLP techniques to learn temporal word representations from diachronic corpora covering a long period of time (Bianchi et al., 2020; Bowern, 2019; Di Carlo et al., 2019; Hamilton et al., 2016b; Tahmasebi, Borin, Jatowt, & Xu, 2019). The advantages of this approach over previous solutions are several: first, it does not require any external resource, such as Swadesh lists, lexica providing estimates of lexical substitutions, and thesauri to estimate polysemy or number of synonyms.…”

“…In order to learn representations that can be used to compute language change, 3 we need diachronic corpora that are divided chronologically in different slices (or bins), such that a different word representation is learned from the language samples in each slice. In order to compare time‐dependent word representations, however, it is necessary to ensure that all word embeddings exist in a vector space defined by the same coordinates (Di Carlo et al., 2019); that is, embeddings from different time slices are aligned. This passage is nontrivial: It is not enough to learn a different embedding space for each slice of a diachronic corpus because such embedding spaces are not necessarily comparable as they have been learned independently.…”