Improved Word Representation Learning with Sememes

Abstract

Sememes are minimum semantic units of word meanings, and the meaning of each word sense is typically composed by several sememes. Since sememes are not explicit for each word, people manually annotate word sememes and form linguistic common-sense knowledge bases. In this paper, we present that, word sememe information can improve word representation learning (WRL), which maps words into a low-dimensional semantic space and serves as a fundamental step for many NLP tasks. The key idea is to utilize word sememes to capture exact meanings of a word within specific contexts accurately. More specifically, we follow the framework of Skip-gram and present three sememe-encoded models to learn representations of sememes, senses and words, where we apply the attention scheme to detect word senses in various contexts. We conduct experiments on two tasks including word similarity and word analogy, and our models significantly outperform baselines. The results indicate that WRL can benefit from sememes via the attention scheme, and also confirm our models being capable of correctly modeling sememe information.

Modeling Semantic Compositionality with Sememe Knowledge

“…We use pretrained word embeddings of MWEs (needed for training in the MWE similarity task) and constituents, which are trained using GloVe (Pennington et al, 2014) on the Sogou-T corpus 4 . We also utilize pretrained sememe embeddings obtained from the results of a sememe-based word representation learning model 5 (Niu et al, 2017).…”

“…HowNet (Dong and Dong, 2003) is a widely acknowledged sememe knowledge base (KB), which defines about 2,000 sememes and uses them to annotate over 100,000 Chinese words together with their English translations. Sememes and HowNet have been successfully utilized in a variety of NLP tasks including sentiment analysis (Dang and Zhang, 2010), word representation learning (Niu et al, 2017), language modeling (Gu et al, 2018), etc.…”

Language Modeling with Sparse Product of Sememe Experts

“…Recently, there are a lot of works concentrating on utilizing sememe knowledge in traditional natural language processing tasks. For example, Niu et al (2017) use sememe knowledge to improve the quality of word embeddings and cope with the problem of word sense disambiguation. apply matrix factorization to predict sememes for words.…”

Cross-lingual Lexical Sememe Prediction

“…Since HowNet was published (Dong and Dong, 2003), it has attracted wide attention of re-searchers. Most of related works focus on applying HowNet to specific NLP tasks (Liu and Li, 2002;Zhang et al, 2005;Sun et al, 2007;Dang and Zhang, 2010;Fu et al, 2013;Niu et al, 2017;Zeng et al, 2018;Gu et al, 2018). To the best of our knowledge, only and Jin et al (2018) conduct studies of augmenting HowNet by recommending sememes for new words.…”