Understanding and bridging the gaps in current GNN performance optimizations

Huang, Kezhao; Zhai, Jidong; Zheng, Zishan; Yi, Youngmin; Shen, Xipeng

doi:10.1145/3437801.3441585

Cited by 60 publications

(32 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Use of Graph Processing in GCN Training: State-of-theart GCN training systems on GPUs [5], [16], [17] typically adopt the vertex-centric programming model to conduct training on input graphs. In GCN training, the number of vertex features can be up to hundreds, which is much larger than that in traditional graph algorithms, which leads to high atomic overheads if the push mode is used.…”

Section: B Graph Processing Systems On Gpusmentioning

confidence: 99%

“…Besides, different from the graph processing systems that generally invoke one software thread to conduct the processing of a vertex, GCN training systems typically divide the features of a vertex into multiple threads for parallel processing. Moreover, for load-balancing, the neighbor grouping technique used in graph processing is also used by recent GCN training systems on GPUs [5], [17]. Similarly, the use of neighbor grouping also leads to atomic overheads on updating the data attached to the vertices as in graph processing systems.…”

Section: B Graph Processing Systems On Gpusmentioning

confidence: 99%

“…By leveraging the technologies of graph processing, GCN training systems on GPUs [5], [8], [16], [17] have exhibited significant performance improvements over the popular deep learning frameworks such as PyTorch [18] and TensorFlow [19]. G 3 [8] proposed to use Gunrock [20], a popular graph processing framework on GPU, to conduct GCN training.…”

Section: Related Workmentioning

confidence: 99%

“…NeuGraph [16] proposes a new programming model Scatter-ApplyEdge-Gather-ApplyVertex by combing the dataflow model and vertex-centric model to support scalable parallel GNN training. The work in [17] accelerates the GNN inference by improving the data locality and load balance as well as reducing redundant computations at the vertex level. GNNAdvisor [5] employs a vertex reorder method (i.e., Rabbit [21]) to improve the data locality, and propose a 2D workload management method to balance the workloads at vertex level and vertex feature level.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Accelerating Backward Aggregation in GCN Training with Execution Path Preparing on GPUs

Xu¹,

Shao²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The emerging Graph Convolutional Network (GCN) has now been widely used in many domains, and it is challenging to improve the efficiencies of applications by accelerating the GCN trainings. For the sparsity nature and exploding scales of input real-world graphs, state-of-the-art GCN training systems (e.g., GNNAdvisor) employ graph processing techniques to accelerate the message exchanging (i.e. aggregations) among the graph vertices. Nevertheless, these systems treat both the aggregation stages of forward and backward propagation phases as all-active graph processing procedures that indiscriminately conduct computation on all vertices of an input graph.In this paper, we first point out that in a GCN training problem with a given training set, the aggregation stages of its backward propagation phase (called as backward aggregations in this paper) can be converted to partially-active graph processing procedures, which conduct computation on only partial vertices of the input graph. By leveraging such a finding, we propose an execution path preparing method that collects and coalesces the data used during backward propagations of GCN training conducted on GPUs. The experimental results show that compared with GNNAdvisor, our approach improves the performance of the backward aggregation of GCN trainings on typical real-world graphs by 1.48x∼5.65x. Moreover, the execution path preparing can be conducted either before the training (during preprocessing) or onthe-fly with the training. When used during preprocessing, our approach improves the overall GCN training by 1.05x∼1.37x. And when used on-the-fly, our approach improves the overall GCN training by 1.03x∼1.35x.Index Terms-graph convolutional network, backward aggregation, graph processing, graph processing unit 1 Trainings in this figure are conducted with GNNAdvisor [5] on an Nvidia Tesla P100 GPU [6], while the input graphs are taken from Table III. In each training, each vertex is given a 128-length random feature vector, and the training set contains 10% vertices of each input graph.

show abstract

Section: B Graph Processing Systems On Gpusmentioning

confidence: 99%

Section: B Graph Processing Systems On Gpusmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Accelerating Backward Aggregation in GCN Training with Execution Path Preparing on GPUs

Xu¹,

Shao²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…FuseGNN (Chen et al, 2020) fuses edge operators to accelerate GNN computation, but it lacks the technique to fuse a vertex-centric operator with an edge-centric one. Huang et al, (Huang et al, 2021) also proposes fusion technique for GNNs, while it cannot handle GNN training because the intermediate data are missing.…”

Section: Introductionmentioning

confidence: 99%

Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective

Zhang¹,

Yu²,

Dai³

et al. 2021

Preprint

View full text Add to dashboard Cite

Graph Neural Networks (GNNs) have been widely used in various domains, and GNNs with sophisticated computational graph lead to higher latency and larger memory consumption. Optimizing the GNN computational graphs suffers from: (1) Redundant neural operator computation. The same data are propagated through the graph structure to perform the same neural operation multiple times in GNNs, leading to redundant computation which accounts for 92.4% of total operators. (2) Inconsistent thread mapping. Efficient thread mapping schemes for vertex-centric and edge-centric operators are different. This inconsistency prohibits operator fusion to reduce memory IO. (3) Excessive intermediate data. For GNN training which is usually performed concurrently with inference, intermediate data must be stored for the backward pass, consuming 91.9% of total memory requirement.To tackle these challenges, we propose following designs to optimize the GNN computational graph from a novel coordinated computation, IO, and memory perspective: (1) Propagation-postponed operator reorganization. We reorganize operators to perform neural operations before the propagation, thus the redundant computation is eliminated. (2) Unified thread mapping for fusion. We propose a unified thread mapping scheme for both vertexand edge-centric operators to enable fusion and reduce IO. (3) Intermediate data recomputation. Intermediate data are recomputed during the backward pass to reduce the total memory consumption. Extensive experimental results on three typical GNN models show that, we achieve up to 2.75× end-to-end speedup, 6.89× less memory IO, and 7.73× less memory consumption over state-of-the-art frameworks.

show abstract