Rethinking graph data placement for graph neural network training on multiple GPUs

Song, Shihui

doi:10.1145/3503221.3508435

Cited by 2 publications

(1 citation statement)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rethinking [86] develops a data-parallel GNN training system, focusing on the data movement optimization among CPU and GPUs. Unlike other systems, it allows GPUs to fetch data from each other by leveraging the high bandwidth among them.…”

Section: Systems On Single Machine With Multi-gpusmentioning

confidence: 99%

Distributed Graph Neural Network Training: A Survey

Shao¹,

Li²,

Gu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph neural networks (GNNs) are a type of deep learning models that learning over graphs, and have been successfully applied in many domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs, since it is able to provide abundant computing resources. However, the dependency of graph structure increases the difficulty of achieving high-efficiency distributed GNN training, which suffers from the massive communication and workload imbalance. In recent years, many efforts have been made on distributed GNN training, and an array of training algorithms and systems have been proposed. Yet, there is a lack of systematic review on the optimization techniques from graph processing to distributed execution. In this survey, we analyze three major challenges in distributed GNN training that are massive feature communication, the loss of model accuracy and workload imbalance. Then we introduce a new taxonomy for the optimization techniques in distributed GNN training that address the above challenges. The new taxonomy classifies existing techniques into four categories that are GNN data partition, GNN batch generation, GNN execution model, and GNN communication protocol. We carefully discuss the techniques in each category. In the end, we summarize existing distributed GNN systems for multi-GPUs, GPU-clusters and CPU-clusters, respectively, and give a discussion about the future direction on scalable GNNs.

show abstract

Section: Systems On Single Machine With Multi-gpusmentioning

confidence: 99%

Distributed Graph Neural Network Training: A Survey

Shao¹,

Li²,

Gu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Comprehensive Evaluation of GNN Training Systems: A Data Management Perspective

Yuan,

Liu,

Zhang

et al. 2024

Proc. VLDB Endow.

View full text Add to dashboard Cite

Many Graph Neural Network (GNN) training systems have emerged recently to support efficient GNN training. Since GNNs embody complex data dependencies between training samples, the training of GNNs should address distinct challenges different from DNN training in data management, such as data partitioning, batch preparation for mini-batch training, and data transferring between CPUs and GPUs. These factors, which take up a large proportion of training time, make data management in GNN training more significant. This paper reviews GNN training from a data management perspective and provides a comprehensive analysis and evaluation of the representative approaches. We conduct extensive experiments on various benchmark datasets and show many interesting and valuable results. We also provide some practical tips learned from these experiments, which are helpful for designing GNN training systems in the future.

show abstract

Rethinking graph data placement for graph neural network training on multiple GPUs

Cited by 2 publications

References 2 publications

Distributed Graph Neural Network Training: A Survey

Distributed Graph Neural Network Training: A Survey

Comprehensive Evaluation of GNN Training Systems: A Data Management Perspective

Contact Info

Product

Resources

About