Simple and automated negative sampling for knowledge graph embedding

Abstract

Negative sampling, which samples negative triplets from non-observed ones in knowledge graph (KG), is an essential step in KG embedding. Recently, generative adversarial network (GAN), has been introduced in negative sampling. By sampling negative triplets with large gradients, these methods avoid the problem of vanishing gradient and thus obtain better performance. However, they make the original model more complex and harder to train. In this paper, motivated by the observation that negative triplets with large gradients are important but rare, we propose to directly keep track of them with the cache. In this way, our method acts as a "distilled" version of previous GAN-based methods, which does not waste training time on additional parameters to fit the full distribution of negative triplets. However, how to sample from and update the cache are two critical questions. We propose to solve these issues by automated machine learning techniques. The automated version also covers GAN-based methods as special cases. Theoretical explanation of NSCaching is also provided, justifying the superior over fixed sampling scheme. Besides, we further extend NSCaching with skip-gram model for graph embedding. Finally, extensive experiments show that our method can gain significant improvements on various Yongqi Zhang 4Paradigm Inc.

NativE: Multi-modal Knowledge Graph Completion in the Wild

“…iii) Negative sampling methods. We employ 3 different negative sampling methods to compare with our method, including KBGAN [4], MANS [60], MMRNS [54]. Among these methods, KBGAN [4] is an adversarial negative sampling method designed for conventional KGC, which applies reinforcement learning to optimize the models.…”

“…Among these methods, KBGAN [4] is an adversarial negative sampling method designed for conventional KGC, which applies reinforcement learning to optimize the models. MANS [60] and MMRNS [54] are two negative sampling strategies designed for MMKGC, which utilize the multi-modal information to enhance the negative sampling process. iv) Numeric-aware KGC methods.…”

“…As mentioned before, NS is an important technology for KGE model training. Therefore, some methods [54,60,64] attempt to enhance the NS process and generate high-quality negative samples by utilizing the multi-modal information. The problem scenarios of existing MMKGC methods are relatively simple and usually consider the text and image modalities, while some of the work just considers the numerical modality.…”

Effects of Locality and Rule Language on Explanations for Knowledge Graph Embeddings

“…In all cases, a scoring function -implemented by minimizing a loss functionguides the training of the embeddings, which are learned to yield high scores for true facts and low scores for false facts. The latter are obtained by corrupting the true facts in the KG -a task of utter importance for the quality of the embeddings [13,36].…”

Efficient Hyper-parameter Search for Knowledge Graph Embedding

“…Using the full set can be defined as the 1VsAll (Lacroix et al, 2018) or kVsAll (Dettmers et al, 2017) according to the positive triplets used. The methods (Cai and Wang, 2018;Zhang et al, 2021) requiring additional models for negative sampling are not considered here. and BCE_sum (Trouillon et al, 2017), to classify the positive and negative triplets as binary classes, or use cross entropy (CE) loss (Lacroix et al, 2018) to classify the positive triplet as the true label over the negative triplets.…”