COSINE: Compressive Network Embedding on Large-scale Information Networks

Zhang, Zhengyan; Yang, Cheng; Liu, Zhiyuan; Sun, Maosong; Fang, Zhichong; Zhang, Bo; Lin, Leyu

doi:10.48550/arxiv.1812.08972

Cited by 4 publications

(5 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Parameter sharing, i.e., model parallelism, is another approach to deal with very large-scale datasets. COSINE and Py-Torch-BigGraph (PGB) [23], [54] utilizes parameter sharing approach. These models generate non-overlapping partitions, having distinct vertices.…”

Section: Large-scale Network Embedding Frameworkmentioning

confidence: 99%

DeHIN: A Decentralized Framework for Embedding Large-scale Heterogeneous Information Networks

Imran¹,

Yin²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modeling heterogeneity by extraction and exploitation of high-order information from heterogeneous information networks (HINs) has been attracting immense research attention in recent times. Such heterogeneous network embedding (HNE) methods effectively harness the heterogeneity of small-scale HINs. However, in the real world, the size of HINs grow exponentially with the continuous introduction of new nodes and different types of links, making it a billion-scale network. Learning node embeddings on such HINs creates a performance bottleneck for existing HNE methods that are commonly centralized, i.e., complete data and the model are both on a single machine. To address large-scale HNE tasks with strong efficiency and effectiveness guarantee, we present Decentralized Embedding Framework for Heterogeneous Information Network (DeHIN) in this paper. In DeHIN, we generate a distributed parallel pipeline that utilizes hypergraphs in order to infuse parallelization into the HNE task. DeHIN presents a context preserving partition mechanism that innovatively formulates a large HIN as a hypergraph, whose hyperedges connect semantically similar nodes. Our framework then adopts a decentralized strategy to efficiently partition HINs by adopting a tree-like pipeline. Then, each resulting subnetwork is assigned to a distributed worker, which employs the deep information maximization theorem to locally learn node embeddings from the partition it receives. We further devise a novel embedding alignment scheme to precisely project independently learned node embeddings from all subnetworks onto a common vector space, thus allowing for downstream tasks like link prediction and node classification. As shown from our experimental results, DeHIN significantly improves the efficiency and accuracy of existing HNE models as well as outperforms the large-scale graph embedding frameworks by efficiently scaling up to large-scale HINs.

show abstract

Section: Large-scale Network Embedding Frameworkmentioning

confidence: 99%

DeHIN: A Decentralized Framework for Embedding Large-scale Heterogeneous Information Networks

Imran¹,

Yin²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Manifold learning models are usually designed for general usage, which may ignore the unique characteristics in the network topology. Most existing network embedding models focus on encapsulating the topology information into the node representations [3,11,18,20,35]. The motivation is that nodes with similar topology structures (e.g., many common neighbors) should be distributed closely in the learned latent space.…”

Section: Related Workmentioning

confidence: 99%

“…Although a lot of network embedding models have been proposed, they usually suffer from high memory usage. Recently some works have focused on learning memory saving embeddings [4,25,35] . DNE [25] learned binary codings as the node embeddings by adding binary subjections into the matrix factorization, which suffered from undesirable embedding quality.…”

Section: Related Workmentioning

confidence: 99%

“…KD [4] was the first work to learn compact embeddings from the pre-learned features, whose performance was limited by the block strategy. Besides, existing models are designed for a specific scenario (pre-learned feature compressing [4] or end-to-end learning [25,35]), but there still lacks a more general framework that can handle both cases.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Hot Compact Network Embedding

Zheng

Wang

et al. 2019

Proceedings of the 28th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Network embedding, as a promising way of the network representation learning, is capable of supporting various subsequent network mining and analysis tasks, and has attracted growing research interests recently. Traditional approaches assign each node with an independent continuous vector, which will cause memory overhead for large networks. In this paper we propose a novel multi-hot compact network embedding framework to effectively reduce memory cost by learning partially shared embeddings. The insight is that a node embedding vector is composed of several basis vectors according to a multi-hot index vector. The basis vectors are shared by different nodes, which can significantly reduce the number of continuous vectors while maintain similar data representation ability. Specifically, we propose a MCNE p model to learn compact embeddings from pre-learned node features. A novel component named compressor is integrated into MCNE p to tackle the challenge that popular back-propagation optimization cannot propagate loss through discrete samples. We further propose an endto-end model MCNE t to learn compact embeddings from the input network directly. Empirically, we evaluate the proposed models over four real network datasets, and the results demonstrate that our proposals can save about 90% of memory cost of network embeddings without significantly performance decline.

show abstract

“…Both of these works only focus on improving embedding quality without improving the scalability. Later,Zhang et al (2018b); Akbas & Aktas (2019) attempt to improve graph embedding scalability by only embedding on the coarsest graph. However, their approaches lack proper refinement methods to generate high-quality embeddings of the original graph Liang et al (2018).…”

mentioning

confidence: 99%

GraphZoom: A multi-level spectral approach for accurate and scalable graph embedding

Deng,

Zhao,

Wang

et al. 2019

Preprint

View full text Add to dashboard Cite

Graph embedding techniques have been increasingly deployed in a multitude of different applications that involve learning on non-Euclidean data. However, existing graph embedding models either fail to incorporate node attribute information during training or suffer from node attribute noise, which compromises the accuracy. Moreover, very few of them scale to large graphs due to their high computational complexity and memory usage. In this paper we propose GraphZoom, a multi-level framework for improving both accuracy and scalability of unsupervised graph embedding algorithms. GraphZoom first performs graph fusion to generate a new graph that effectively encodes the topology of the original graph and the node attribute information. This fused graph is then repeatedly coarsened into a much smaller graph by merging nodes with high spectral similarities. GraphZoom allows any existing embedding methods to be applied to the coarsened graph, before it progressively refine the embeddings obtained at the coarsest level to increasingly finer graphs. We have evaluated our approach on a number of popular graph datasets for both transductive and inductive tasks. Our experiments show that GraphZoom increases the classification accuracy and significantly reduces the run time compared to state-of-the-art unsupervised embedding methods.

show abstract

COSINE: Compressive Network Embedding on Large-scale Information Networks

Cited by 4 publications

References 42 publications

DeHIN: A Decentralized Framework for Embedding Large-scale Heterogeneous Information Networks

DeHIN: A Decentralized Framework for Embedding Large-scale Heterogeneous Information Networks

Multi-Hot Compact Network Embedding

GraphZoom: A multi-level spectral approach for accurate and scalable graph embedding

Contact Info

Product

Resources

About