We present a state-of-the-art algorithm for measuring the semantic similarity of word pairs using novel combinations of word embeddings, WordNet, and the concept dictionary 4lang. We evaluate our system on the SimLex-999 benchmark data. Our top score of 0.76 is higher than any published system that we are aware of, well beyond the average inter-annotator agreement of 0.67, and close to the 0.78 average correlation between a human rater and the average of all other ratings, suggesting that our system has achieved nearhuman performance on this benchmark.
IntroductionWe present a hybrid system for measuring the semantic similarity of word pairs. The system relies both on standard word embeddings, the WordNet database, and features derived from the 4lang concept dictionary, a set of concept graphs built from entries in monolingual dictionaries of English. 4lang-based features improve the performance of systems using only word embeddings and/or WordNet, our top configurations achieve state-of-the-art results on the SimLex-999 data, which has recently become a popular benchmark of word similarity metrics.In Section 1 we summarize earlier work on measuring word similarity and review the latest results achieved on the SimLex-999 data. Section 2 describes our experimental setup, Sections 2.1 and 2.2 documents the features obtained using word embeddings and WordNet. In Section 3 we briefly introduce the 4lang resources and the formalism it uses for encoding the meaning of words as directed graphs of concepts, then document our efforts to develop novel 4lang-based similarity features. Besides improving the performance of existing systems for measuring word similarity, the goal of the present project is to examine the potential of 4lang representations in representing non-trivial lexical relationships that are beyond the scope of word embeddings and standard linguistic ontologies.Section 4 presents our results and provides rough error analysis. Section 5 offers some conclusions and plans for future work. All software presented in this paper is available for download under an MIT license at