DAWE: A Double Attention-Based Word Embedding Model with Sememe Structure Information

Abstract

Word embedding is an important reference for natural language processing tasks, which can generate distribution presentations of words based on many text data. Recent evidence demonstrates that introducing sememe knowledge is a promising strategy to improve the performance of word embedding. However, previous works ignored the structure information of sememe knowledges. To fill the gap, this study implicitly synthesized the structural feature of sememes into word embedding models based on an attention mechanism. Specifically, we propose a novel double attention word-based embedding (DAWE) model that encodes the characteristics of sememes into words by a “double attention” strategy. DAWE is integrated with two specific word training models through context-aware semantic matching techniques. The experimental results show that, in word similarity task and word analogy reasoning task, the performance of word embedding can be effectively improved by synthesizing the structural information of sememe knowledge. The case study also verifies the power of DAWE model in word sense disambiguation task. Furthermore, the DAWE model is a general framework for encoding sememes into words, which can be integrated into other existing word embedding models to provide more options for various natural language processing downstream tasks.

Special Issue on “Natural Language Processing: Emerging Neural Approaches and Applications”

“…Trustworthy and explainable artificial intelligence: Two contributions [40,41] considered "sememe", the smallest semantic unit for describing real-world concepts, which improve the interpretability of NLP systems. In particular, the study in [40] proposed a novel model to improve the performance of sememe prediction by introducing synonyms.…”

“…In particular, the study in [40] proposed a novel model to improve the performance of sememe prediction by introducing synonyms. On the other hand, the work in [41] implicitly synthesized the structural features of sememes into word embedding models through an attention mechanism. The work proposes a novel double attention word-based embedding (DAWE) model that encodes the characteristics of sememes into words with a "double attention" strategy.…”