One of these words is not like the other: a reproduction of outlier identification using non-contextual word representations

Abstract

Word embeddings are an active topic in the NLP research community. State-of-the-art neural models achieve high performance on downstream tasks, albeit at the cost of computationally expensive training. Cost aware solutions require cheaper models that still achieve good performance. We present several reproduction studies of intrinsic evaluation tasks that evaluate non-contextual word representations in multiple languages. Furthermore, we present 50-8-8, a new data set for the outlier identification task, which avoids limitations of the original data set, such as ambiguous words, infrequent words, and multi-word tokens, while increasing the number of test cases. The data set is expanded to contain semantic and syntactic tests and is multilingual (English, German, and Italian).We provide an in-depth analysis of word embedding models with a range of hyperparameters. Our analysis shows the suitability of different models and hyper-parameters for different tasks and the greater difficulty of representing German and Italian languages.

Large Scale Substitution-based Word Sense Induction

“…In each evaluation round, one outlier is added to the in-group items, and the algorithm is tasked with finding the outlier. Existing outlier detection datasets either did not explicitly target sense-ambiguous words (8-8-8 (Camacho-Collados and Navigli, 2016), WikiSem500 (Blair et al, 2016)) or explicitly removed ambiguous words altogether (25-8-8-sem (Brink Andersen et al, 2020)).…”

“…The starting point for our dataset is 25-8-8-Sem (Brink Andersen et al, 2020). This dataset contains 25 test groups, each with 8 in-group elements and 8 outliers, resulting in 200 unique test cases.…”