NetSMF: Large-Scale Network Embedding as Sparse Matrix Factorization

Abstract

We study the problem of large-scale network embedding, which aims to learn latent representations for network mining applications. Previous research shows that 1) popular network embedding benchmarks, such as DeepWalk, are in essence implicitly factorizing a matrix with a closed form, and 2) the explicit factorization of such matrix generates more powerful embeddings than existing methods. However, directly constructing and factorizing this matrix-which is dense-is prohibitively expensive in terms of both time and space, making it not scalable for large networks.In this work, we present the algorithm of large-scale network embedding as sparse matrix factorization (NetSMF). NetSMF leverages theories from spectral sparsification to efficiently sparsify the aforementioned dense matrix, enabling significantly improved efficiency in embedding learning. The sparsified matrix is spectrally close to the original dense one with a theoretically bounded approximation error, which helps maintain the representation power of the learned embeddings. We conduct experiments on networks of various scales and types. Results show that among both popular benchmarks and factorization based methods, NetSMF is the only method that achieves both high efficiency and effectiveness. We show that NetSMF requires only 24 hours to generate effective embeddings for a large-scale academic collaboration network with tens of millions of nodes, while it would cost DeepWalk months and is computationally infeasible for the dense matrix factorization solution. The source code of NetSMF is publicly available 1 .

Knowledge Graph Completion: A Review

“…The practicality is greatly restricted. In order to retain the performance of the NetMF algorithm and reduce the requirements for computing resources, NetSMF proposes to sparse the NetMF matrix [88], and keep the spectrum of the sparse matrix close to the original matrix, and then decompose the sparse matrix. Experiments show that this method can effectively improve the computational efficiency of the algorithm while retaining the spectrum information of the network.…”

Fast and Accurate Network Embeddings via Very Sparse Random Projection

“…Most previous matrix-based network embedding methods emphasize on the importance of high-order proximity but skip the element-wise normalization step for either better scalability or ease of analysis [25,26,38]. Is normalization of the node similarity matrix important?…”

“…However, GraRep is not scalable due to the high time complexity of both raising the transition matrix A to higher powers and taking the element-wise logarithm of A k , which is a dense n × n matrix. A few recent work thus propose to speed up the construction of such a node similarity matrix [25,26,39], which are inspired by the spectral graph theory. The basic idea is that if the top-h eigendecomposition of A is given by A = U h Λ h U ⊤ h , then A k can be approximated with U h Λ k h U ⊤ h [26,39].…”

“…A recent method [25] proposes to sparsify A k for better scalability. However, the similarity matrix is still dense even after the sparsification: for a graph with n = 10 6 nodes and m = 10 7 edges, the number of entries in the sparsified matrix can be as high as 1.4 × 10 11 [25].…”

Representation Learning for Attributed Multiplex Heterogeneous Network

“…DeepWalk [27], LINE [35], and node2vec [10] are pioneering works that introduce deep learning techniques into network analysis to learn node embeddings. NetMF [29] gives a theoretical analysis of equivalence for the different network embedding algorithms, and later NetSMF [28] gives a scalable solution via sparsification. Nevertheless, they were designed to handle only the homogeneous network with single-typed nodes and edges.…”