Binarized Knowledge Graph Embeddings

A Greedy Bit-flip Training Algorithm for Binarized Knowledge Graph Embeddings

“…The figure shows greedy bit flipping (Bit-flip B-CP) requires a much smaller number of training epochs to converge than HSTE-based training (HSTE B-CP). For both datasets, the best MRR (Ruffinelli et al, 2020), (Kishimoto et al, 2019b) and (Li et al, 2020), respectively. The memory consumption figures for these models are estimated from the reported number of parameters.…”

“…Thus, vectors a i: , b j: , c k: can be regarded as the D-dimensional vectors representing the subject entity e i , object entity e j , and relation label r k , respectively. The B-CP decomposition of a knowledge graph (Kishimoto et al, 2019b) differs from the standard CP, in that X is decomposed in terms of binary vectors…”

“…To deal with knowledge graphs with more than a million entities, more lightweight models are desirable for faster processing and to reduce memory consumption, as AI applications on mobile devices are becoming more common. Kishimoto et al (2019b) proposed a binarized KGE model B-CP, wherein all vector components are binarized, allowing them to be stored compactly with bitwise representation. Despite the reduced memory consumption by more than a magnitude, B-CP performed as well as the existing real-valued KGE models on benchmark tasks.…”

“…During the training of B-CP, however, realvalued embeddings are maintained and are quantized at each training step (Kishimoto et al, 2019b). The loss function is computed with respect to the quantized vectors, but stochastic gradient descent is performed on the real vectors with the help of Hinton's "straight-through" estimator (HSTE) (Bengio et al, 2013).…”

Binarized Embeddings for Fast, Space-Efficient Knowledge Graph Completion

“…Score function, time complexity of score computation, and full expressiveness of KGE models1 ModelScore function Complexity Fully expressiveTransE − ae i + br k − ae j O(D) RESCAL a T e i Br k ae j O(D 2 ) DistMult (ae i • ae j ) T br k O(D) CP (ae i • be j ) T cr k O(D) B-CP −h(āe i , XNOR(be j ,cr k )) O(D)…”