Deep Neural Models of Semantic Shift

Abstract

Diachronic distributional models track changes in word use over time. In this paper, we propose a deep neural network diachronic distributional model. Instead of modeling lexical change via a time series as is done in previous work, we represent time as a continuous variable and model a word's usage as a function of time. Additionally, we have created a novel synthetic task, which quantitatively measures how well a model captures the semantic trajectory of a word over time. Finally, we explore how well the derivatives of our model can be used to measure the speed of lexical change.

From Insanely Jealous to Insanely Delicious: Computational Models for the Semantic Bleaching of English Intensifiers

“…Developments in computational semantics and availability of large diachronic corpora have renewed interest in studying historical semantic change. Recent work has moved away from documenting and qualitatively categorizing types of changes (Bréal, 1964;Stern, 1931) to focus on detecting semantic shifts (Gulordava and Baroni, 2011;Rosenfeld and Erk, 2018;Frermann and Lapata, 2016;Mitra et al, 2014;Kulkarni et al, 2015), distinguishing gradual linguistic drifts from cultural ones (Hamilton et al, 2016a) and assessing laws of change (Hamilton et al, 2016b;Dubossarsky et al, 2017;Xu and Kemp, 2015;Ramiro et al, 2018;Luo and Xu, 2018).…”

Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View

“…Bamler and Mandt (2017) propose to use dynamic word embeddings trained jointly over all times periods. Instead of modeling lexical change via time series, Rosenfeld and Erk (2018) represent time as a continuous variable and model a word's usage as a function of time. Yin et al (2018) propose global anchor method for detecting linguistic shifts and domain adaptation.…”

Diachronic Cross-modal Embeddings

“…In Uricchio et al [29], the value of temporal information for the tasks of image annotation and retrieval, such as tag frequency, is recognised. In order to model the temporal behaviour of data, embeddings must retain temporal correlations [2,9,15,24,27,38]. The challenge resides in capturing such correlations and incorporating them in cross-modal embeddings.…”

“…Conversely, with large bins only coarse grained representations can be obtained. To overcome this, Rosenfeld and Erk [24] recently proposed a continuous approach, in which time is taken as a continuous variable. The model learns an embedding for each word w at each time instant t. Our work goes in this direction, however two aspects invalidate the use of existing word diachronic models: a) unlike words, that are predominant across time instants, each instance is posted only once, invalidating existing alignment strategies, b) in the cross-modal scenario two modalities need to be aligned instead of only one.…”

“…Accordingly, textual descriptions of images reflect the way humans refer to each image at a given point in time, thus being susceptible to change over time, either due to the occurrence of external events [16] or simply by word meaning change across time [9,38]. Recently, word embedding models that capture language evolution over time (referred as distributional diachronic models) have been proposed [9,24,38]. In our setting we seek to obtain representations that capture the evolution of visual and textual correlations over time.…”