SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection

Abstract

Lexical Semantic Change detection, i.e., the task of identifying words that change meaning over time, is a very active research area, with applications in NLP, lexicography, and linguistics. Evaluation is currently the most pressing problem in Lexical Semantic Change detection, as no gold standards are available to the community, which hinders progress. We present the results of the first shared task that addresses this gap by providing researchers with an evaluation framework and manually annotated, high-quality datasets for English, German, Latin, and Swedish. 33 teams submitted 186 systems, which were evaluated on two subtasks.

Three-part diachronic semantic change dataset for Russian

“…The inter-rater agreement statistics and the number of judgments in each RuShiftEval subset are shown in Table 1. The agreement is on par with other semantic change annotation efforts: (Schlechtweg et al, 2020) report Spearman correlations ranging from 0.58 to 0.69, (Rodina and Kutuzov, 2020) report Krippendorff's α ranging from 0.51 to 0.53. 3 Each subset was annotated by about 100 human raters, more or less uniformly 'spread' across annotation instances, with the only constraint being that each instance must be annotated by three different persons.…”

“…A similar approach was used for the Se-mEval'20 shared task on semantic change detection (Schlechtweg et al, 2020): annotators labeled pairs of sentences, where some pairs belonged to the same periods and some to different ones. This annotation resulted in a diachronic word usage graph, which was then clustered to obtain sepa-rate word senses and their distributions between time periods (Schlechtweg et al, 2021).…”

“…words that changed their meaning between either pair of periods; (ii) not all words in the dataset actually changed their meaning. We followed the same procedure as in (Kutuzov and Kuzmenko, 2018;Rodina and Kutuzov, 2020;Schlechtweg et al, 2020): first, select changing words, and then augment them with fillers, i.e. random words following similar frequency distribution across three time periods.…”

Semantic Journeys: Quantifying Change in Emoji Meaning from 2012-2018

“…Research into computational methods for the automatic detection of semantic change is currently an active area of research. Researchers have proposed a range of different approaches, from topic-based models (Cook et al 2014;Lau et al 2014;Frermann and Lapata 2016), to graph-based models (Mitra et al 2015;Tahmasebi and Risse 2017) (Hamilton, Leskovec, and Jurafsky 2016a;Perrone et al 2019;Schlechtweg et al 2020), with some notable exceptions, such as Shoemark et al (2019), who perform a systematic evaluation of embedding-based methods for short-term semantic change detection using Twitter data from 2011 to 2017. All methods proposed so far have dealt with semantic change of words, and to our knowledge no work has yet been done on applying this research to emoji specifically.…”

Lexical semantic change for Ancient Greek and Latin

“…Most work in this area focusses on simply detecting the occurrence of semantic change, while Frermann and Lapata (2016)'s system, SCAN, takes into account synchronic polysemy and models how the different word senses evolve across time. More recently French has been further tackled by Jawahar andSeddah (2019), Frossard et al (2020) and Montariol and Allauzen (2020), and German has been the focus of extensive work (Schlechtweg et al, 2017(Schlechtweg et al, , 2018(Schlechtweg et al, , 2020.…”