Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective

Abstract

Graph Neural Networks (GNNs) have been widely used in various domains, and GNNs with sophisticated computational graph lead to higher latency and larger memory consumption. Optimizing the GNN computational graphs suffers from: (1) Redundant neural operator computation. The same data are propagated through the graph structure to perform the same neural operation multiple times in GNNs, leading to redundant computation which accounts for 92.4% of total operators. (2) Inconsistent thread mapping. Efficient thread mapping schemes for vertex-centric and edge-centric operators are different. This inconsistency prohibits operator fusion to reduce memory IO. (3) Excessive intermediate data. For GNN training which is usually performed concurrently with inference, intermediate data must be stored for the backward pass, consuming 91.9% of total memory requirement.To tackle these challenges, we propose following designs to optimize the GNN computational graph from a novel coordinated computation, IO, and memory perspective: (1) Propagation-postponed operator reorganization. We reorganize operators to perform neural operations before the propagation, thus the redundant computation is eliminated. (2) Unified thread mapping for fusion. We propose a unified thread mapping scheme for both vertexand edge-centric operators to enable fusion and reduce IO. (3) Intermediate data recomputation. Intermediate data are recomputed during the backward pass to reduce the total memory consumption. Extensive experimental results on three typical GNN models show that, we achieve up to 2.75× end-to-end speedup, 6.89× less memory IO, and 7.73× less memory consumption over state-of-the-art frameworks.

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis

“…Instead, they target GNN computations from scratch, focusing on GNN-specific workload characteristics and design decisions [113], [171], [172], [215]. For example, Zhang et al [238] analyze the computational graph of GNNs, and propose optimizations tailored specifically for GNNs.…”

“…While most systems fuse operators within a single GNN layer, QGTC offers operator fusion also across different GNN layers [214]. Other interesting schemes include operator reorganization [238], in which one ensures that operators first perform neural operations (e.g., MLP) and only then the updated feature vectors are propagated. This limits redundant computation.…”

“…Many other schemes exist. For example, Zhang et al [238] reduce memory consumption (intermediate data are recomputed in the backward pass, instead of being cached), while He et al [102] incorporate serverless computing [67] for more efficient GNNs.…”